<div dir="ltr">In addition to the above problem . oversubscription is NO then according to the document.so in this scenario even if resources are available it is ot accepting the job from other partition. Even i made the same priority for both the partition but it didn't help. Any Suggestion here.<div><br><div>Slurm Workload Manager - Sharing Consumable Resources
<div class="gmail-container gmail-container--main"><div class="gmail-content"><div class="gmail-section"><div class="gmail-container"><table border="1" cellspacing="1" cellpadding="3"><tbody><tr><td>Two <code>OverSubscribe=NO</code> partitions assigned the same set of nodes</td><td>Jobs from either partition will be assigned to all available consumable resources. No consumable resource will be shared. One node could have 2 jobs running on it, and each job could be from a different partition.</td></tr></tbody></table></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 31, 2020 at 4:34 PM navin srivastava <<a href="mailto:navin.altair@gmail.com">navin.altair@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi ,<div><br></div><div>have an issue with the resource allocation.</div><div><br></div><div>In the environment have partition like below:</div><div><br></div><div>PartitionName=small_jobs Nodes=Node[17,20] Default=NO MaxTime=INFINITE State=UP Shared=YES Priority=8000 <br>PartitionName=large_jobs Nodes=Node[17,20] Default=NO MaxTime=INFINITE State=UP Shared=YES Priority=100 <br></div><div><br></div><div>Also the node allocated with less cpu and lot of cpu resources available</div><div><br></div><div>NodeName=Node17 Arch=x86_64 CoresPerSocket=18<br> CPUAlloc=4 CPUErr=0 CPUTot=36 CPULoad=4.09<br> AvailableFeatures=K2200<br> ActiveFeatures=K2200<br> Gres=gpu:2<br> NodeAddr=Node1717 NodeHostName=Node17 Version=17.11<br> OS=Linux 4.12.14-94.41-default #1 SMP Wed Oct 31 12:25:04 UTC 2018 (3090901)<br> RealMemory=1 AllocMem=0 FreeMem=225552 Sockets=2 Boards=1<br> State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br> Partitions=small_jobs,large_jobs<br> BootTime=2020-03-21T18:56:48 SlurmdStartTime=2020-03-31T09:07:03<br> CfgTRES=cpu=36,mem=1M,billing=36<br> AllocTRES=cpu=4<br> CapWatts=n/a<br> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br></div><div><br></div><div>there is no other job in small_jobs partition but several jobs are in pending in the large_jobs and the resources are available but jobs are not going through.</div><div><br></div><div>one of the job pening output is:</div><div><br></div><div>scontrol show job 1250258<br> JobId=1250258 JobName=import_workflow<br> UserId=m209767(100468) GroupId=oled(4289) MCS_label=N/A<br> Priority=363157 Nice=0 Account=oledgrp QOS=normal<br> JobState=PENDING Reason=Priority Dependency=(null)<br> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0<br> RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A<br> SubmitTime=2020-03-28T22:00:13 EligibleTime=2020-03-28T22:00:13<br> StartTime=2070-03-19T11:59:09 EndTime=Unknown Deadline=N/A<br> PreemptTime=None SuspendTime=None SecsPreSuspend=0<br> LastSchedEval=2020-03-31T12:58:48<br> Partition=large_jobs AllocNode:Sid=deda1x1466:62260<br> ReqNodeList=(null) ExcNodeList=(null)<br> NodeList=(null)<br> NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*<br> TRES=cpu=1,node=1<br> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br> MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0<br> Features=(null) DelayBoot=00:00:00<br> Gres=(null) Reservation=(null)<br> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)</div><div><br></div><div>this is my slurm.conf file for scheduling.</div><div><br></div><div><br></div><div>SchedulerType=sched/builtin<br>#SchedulerParameters=enable_user_top<br>SelectType=select/cons_res<br>#SelectTypeParameters=CR_Core_Memory<br>SelectTypeParameters=CR_Core<br> <br></div><div><br></div><div>Any idea why the job is not going for execution if cpu cores are avaiable.</div><div><br></div><div>Also would like to know if any jobs are running on a particular node and if i restart the Slurmd service then in what scenario my job will get killed. Generally it should not kill the job.</div><div><br></div><div>Regards</div><div>Navin.</div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div>