<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:monospace">> You might want to look at BatchStartTimeout Parameter</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">I've got that set to 300 seconds. Every so often one node here and there won't start and gets "ResumeTimeoutExceeded", but we're not seeing those associated with this situation (i.e. nothing in that state in this particular partition)</div><div class="gmail_default" style="font-family:monospace"><br></div><div class="gmail_default" style="font-family:monospace">> what state are the Slurm power saving powered-down nodes in when powered-down? From man pages sounds like should be idle with a flag indicated Slurm powered them down???</div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">Yep- these nodes show "</font><span style="font-family:monospace">State=IDLE+CLOUD+POWER". sinfo shows these nodes with the tilde. All the idle nodes are usable and start when un-blocked workload shows up (i.e. jobs that don't violate a limit).</span></div><div class="gmail_default"><span style="font-family:monospace"><br></span></div><div class="gmail_default"><span style="font-family:monospace">> </span><font face="monospace"> Can you manually force one of the powered-down nodes to power up, and see if the large job gets assigned to it?</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">Negative. Just sits idle with "AssocGrpLimit".</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">The search continues- thanks</font></div><div class="gmail_default"><font face="monospace"><br></font></div><div class="gmail_default"><font face="monospace">M</font></div><div class="gmail_default"><span style="font-family:monospace"><br></span></div><input name="virtru-metadata" type="hidden" value="{"email-policy":{"state":"closed","expirationUnit":"days","disableCopyPaste":false,"disablePrint":false,"disableForwarding":false,"enableNoauth":false,"expires":false,"isManaged":false},"attachments":{},"compose-id":"7","compose-window":{"secure":false}}"></div></div></div></div></div><br><div class="gmail_quote" style=""><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 2:34 PM Thomas M. Payerle <<a href="mailto:payerle@umd.edu">payerle@umd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>I am not very familiar with the Slurm power saving stuff. You might want to look at BatchStartTimeout Parameter (See e.g. <a href="https://slurm.schedmd.com/power_save.html" target="_blank">https://slurm.schedmd.com/power_save.html</a>)</div><div><br></div><div>Otherwise, what state are the Slurm power saving powered-down nodes in when powered-down? From man pages sounds like should be idle with a flag indicated Slurm</div><div>powered them down???</div><div><br></div><div>As an experiment, when you have the 6 core jobs blocked by the account 300 core limit, presumable only remaining "idle" nodes were powered down by Slurm power saving stuff. Can you manually force one of the powered-down nodes to power up, and see if the large job gets assigned to it?</div><div>Is it possible Slurm is not able to power up the nodes?<br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 4:45 PM Michael Gutteridge <<a href="mailto:michael.gutteridge@gmail.com" target="_blank">michael.gutteridge@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:monospace">> You have not provided enough information (cluster configuration, job information, etc) to diagnose what accounting policy is being violated.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">Yeah, sorry. I'm trying to balance the amount of information and likely skewed too concise 8-/</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">The partition looks like:</div><div style="font-family:monospace"><br></div><div><div><font face="monospace">PartitionName=largenode</font></div><div><font face="monospace"> AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL</font></div><div><font face="monospace"> AllocNodes=ALL Default=NO QoS=lg_node_default</font></div><div><font face="monospace"> DefaultTime=3-00:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO</font></div><div><font face="monospace"> MaxNodes=UNLIMITED MaxTime=7-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=18</font></div><div><font face="monospace"> Nodes=lg_nodeg[0-103],lg_nodeh[0-34]</font></div><div><font face="monospace"> PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=YES:4</font></div><div><font face="monospace"> OverTimeLimit=NONE PreemptMode=OFF</font></div><div><font face="monospace"> State=UP TotalCPUs=2432 TotalNodes=139 SelectTypeParameters=NONE</font></div><div><font face="monospace"> JobDefaults=(null)</font></div><div><font face="monospace"> DefMemPerNode=UNLIMITED MaxMemPerNode=245859</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">The partition QOS (lg_node_default) had no limits configured- as indicated, I've since added a "MaxTRESPU" to limit per-user CPU utilisation to get jobs running again.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">The nodes in this partition are large enough to satisfy both the larger and smaller jobs:</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">NodeName=lg_nodeg1 Arch=x86_64 CoresPerSocket=1 </font></div><div><font face="monospace"> CPUAlloc=18 CPUTot=18 CPULoad=3.00</font></div><div><font face="monospace"> AvailableFeatures=c5.9xlarge</font></div><div><font face="monospace"> ActiveFeatures=c5.9xlarge</font></div><div><font face="monospace"> Gres=(null)</font></div><div><font face="monospace"> NodeAddr=<a href="http://lg_nodeg1.fhcrc.org" target="_blank">lg_nodeg1.fhcrc.org</a> NodeHostName=lg_nodeg1 Port=0 Version=18.08</font></div><div><font face="monospace"> OS=Linux 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 </font></div><div><font face="monospace"> RealMemory=70348 AllocMem=0 FreeMem=25134 Sockets=18 Boards=1</font></div><div><font face="monospace"> State=ALLOCATED+CLOUD ThreadsPerCore=1 TmpDisk=7924 Weight=40 Owner=N/A MCS_label=N/A</font></div><div><font face="monospace"> Partitions=largenode </font></div><div><font face="monospace"> BootTime=2019-02-20T07:58:22 SlurmdStartTime=2019-02-20T07:58:38</font></div><div><font face="monospace"> CfgTRES=cpu=18,mem=70348M,billing=18</font></div><div><font face="monospace"> AllocTRES=cpu=18</font></div><div><font face="monospace"> CapWatts=n/a</font></div><div><font face="monospace"> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</font></div><div><font face="monospace"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">NodeName=lg_nodeh1 Arch=x86_64 CoresPerSocket=1 </font></div><div><font face="monospace"> CPUAlloc=12 CPUTot=16 CPULoad=2.02</font></div><div><font face="monospace"> AvailableFeatures=r4.8xlarge</font></div><div><font face="monospace"> ActiveFeatures=r4.8xlarge</font></div><div><font face="monospace"> Gres=(null)</font></div><div><font face="monospace"> NodeAddr=<a href="http://lg_nodeh1.fhcrc.org" target="_blank">lg_nodeh1.fhcrc.org</a> NodeHostName=lg_nodeh1 Port=0 Version=18.08</font></div><div><font face="monospace"> OS=Linux 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 </font></div><div><font face="monospace"> RealMemory=245853 AllocMem=0 FreeMem=147943 Sockets=16 Boards=1</font></div><div><font face="monospace"> State=MIXED+CLOUD ThreadsPerCore=1 TmpDisk=7924 Weight=80 Owner=N/A MCS_label=N/A</font></div><div><font face="monospace"> Partitions=largenode </font></div><div><font face="monospace"> BootTime=2019-02-27T01:35:35 SlurmdStartTime=2019-02-27T01:35:47</font></div><div><font face="monospace"> CfgTRES=cpu=16,mem=245853M,billing=16</font></div><div><font face="monospace"> AllocTRES=cpu=12</font></div><div><font face="monospace"> CapWatts=n/a</font></div><div><font face="monospace"> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</font></div><div><font face="monospace"> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</font></div><div><font face="monospace"><br></font></div><div><font face="monospace">The limit that is in play is on the account:</font></div><div><font face="monospace"><br></font></div><div><font face="monospace"> Account GrpJobs GrpNodes GrpCPUs GrpMem GrpSubmit </font></div><div><font face="monospace">---------- ------- -------- -------- ------- --------- </font></div><div><font face="monospace"> account1 300 </font></div><div style="font-family:monospace"><br></div><div style="font-family:monospace">Some possibly relevant slurm.conf parameters:</div><div style="font-family:monospace"><br></div><div><div style="font-family:monospace">AccountingStorageType=accounting_storage/slurmdbd</div><div style="font-family:monospace">AccountingStorageEnforce=limits,qos</div><div><div><font face="monospace">FastSchedule=0</font></div><div><font face="monospace">SchedulerType=sched/backfill</font></div><div><font face="monospace">SchedulerParameters=bf_resolution=360,defer,bf_continue,bf_max_job_user=10,bf_window=10080</font></div><div><font face="monospace">SelectType=select/cons_res</font></div><div><font face="monospace">SelectTypeParameters=CR_CPU</font></div><div><font face="monospace">PreemptMode=OFF</font></div><div><font face="monospace">PreemptType=preempt/none</font></div><div><font face="monospace">PriorityType=priority/multifactor</font></div><div><font face="monospace">PriorityDecayHalfLife=1-00:00:00</font></div><div><font face="monospace">PriorityMaxAge=1-00:00:00</font></div><div><font face="monospace">PriorityWeightAge=10</font></div><div><font face="monospace">PriorityWeightFairshare=100000000</font></div><div><font face="monospace">PriorityWeightQOS=1000000</font></div></div><div style="font-family:monospace"><br></div><div style="font-family:monospace">and finally the power-management:</div><div style="font-family:monospace"><br></div><div style="font-family:monospace"><div>SuspendProgram=/var/lib/slurm-llnl/suspend</div><div>SuspendTime=300</div><div>SuspendRate=10</div><div>ResumeProgram=/var/lib/slurm-llnl/resume</div><div>ResumeRate=10</div><div>ResumeTimeout=300</div><div><br></div><div>There's no logic in the suspend/resume scripts- these simply start and stop nodes according to what slurmctld says. I don't know exactly what logic the controller uses to start or stop nodes, but I do know it isn't attempting to start nodes to satisfy the larger job.</div><div><br></div><div>> JobId=2210784 delayed for accounting policy is likely the key as it indicates the job is currently unable to run, so the lower priority smaller job bumps ahead of it.</div><div><br></div><div>Yeah, that's exactly what I think is happening. In the on-prem cluster the backfill scheduler creates a priority reservation for that higher-priority job which keeps those from getting starved. However, out in the cloud cluster that doesn't seem to happen.</div><div><br></div><div>Thanks for looking at the problem</div><div><br></div><div> - Michael</div></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 12:54 PM Thomas M. Payerle <<a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The "<span class="gmail_default" style="font-family:monospace"></span>JobId=2210784 delayed for accounting policy is likely the key as it indicates the job is currently unable to run, so the lower priority smaller job bumps ahead of it.</div><div>You have not provided enough information (cluster configuration, job information, etc) to diagnose what accounting policy is being violated. Like you, I suspect that this is happening due to power management and powered-down nodes (I am not experienced with sending jobs to the cloud) --- what is the policy for starting the powered down nodes? I can also see issues due to the delay in starting the powered down nodes; the scheduler starts looking at 2210784, are not enough idle, running nodes to launch it, maybe it tells some nodes to spin up, but by time they spin up it already assigned the previously up and idle nodes to the smaller job.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 3:33 PM Michael Gutteridge <<a href="mailto:michael.gutteridge@gmail.com" target="_blank">michael.gutteridge@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:monospace">I've run into a problem with a cluster we've got in a cloud provider- hoping someone might have some advice.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">The problem is that I've got a circumstance where large jobs _never_ start... or more correctly, that large-er jobs don't start when there are many smaller jobs in the partition. In this cluster, accounts are limited to 300 cores. One user has submitted a couple thousand jobs that each use 6 cores. These queue up, start nodes, and eventually all 300 cores in the limit are busy and the remaining jobs are held with "AssocGrpCpuLimit". All as expected.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">Then another user submits a job requesting 16 cores. This one, too, gets held with the same reason. However, that larger job never starts even if it has the highest priority of jobs in this account (I've set it manually and by using nice).</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">What I see in the sched.log is:</div><div style="font-family:monospace"><br></div><div><div style="font-family:monospace">sched: [2019-02-25T16:00:14.940] Running job scheduler</div><div style="font-family:monospace">sched: [2019-02-25T16:00:14.941] JobId=2210784 delayed for accounting policy</div><div style="font-family:monospace">sched: [2019-02-25T16:00:14.942] JobId=2203130 initiated</div><div style="font-family:monospace">sched: [2019-02-25T16:00:14.942] Allocate JobId=2203130 NodeList=node1 #CPUs=6 Partition=largenode</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">In this case, 2210784 is the job requesting 16 cores and 2203130 is one of the six core jobs. This seems to happen with either the backfill or builtin scheduler. I suspect what's happening is that when one of the smaller jobs completes, the scheduler first looks at the higher-priority large job, determines that it cannot run because of the constraint, looks at the next job in the list, determines that it can run without exceeding the limit, and then starts that job. In this way, the larger job isn't started until all of these smaller jobs complete.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">I thought that switching to the builtin scheduler would fix this, but as slurm.conf(5) indicates:</div><div style="font-family:monospace"><br></div><div><div><font face="monospace">> An </font><span style="font-family:monospace">exception is made for jobs that can not run due </span></div><div><span style="font-family:monospace">> to partition </span><span style="font-family:monospace">constraints (e.g. the time limit) or </span></div><div><span style="font-family:monospace">> down/drained nodes. In </span><span style="font-family:monospace">that case, lower priority </span></div><div><span style="font-family:monospace">> jobs can be initiated and not </span><span style="font-family:monospace">impact the higher </span></div><div><span style="font-family:monospace">> priority job.</span></div></div><div style="font-family:monospace"><br></div><div style="font-family:monospace">I suspect one of these exceptions is being triggered- the limit is in the job's association, so I don't think it's a partition constraint. I don't have this problem with the on-premises cluster, so I suspect it's something to do with power management and the state of powered-down nodes.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">I've sort-of worked around this by setting a per-user limit lower than the per-account limit, but that doesn't address any situation where a single user submits large and small jobs and does lead to some other problems in other groups, so it's not a long-term solution.</div><div style="font-family:monospace"><br></div><div style="font-family:monospace">Thanks for having a look</div><div style="font-family:monospace"><br></div><div style="font-family:monospace"> - Michael</div><div style="font-family:monospace"><br></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_8508270196494718028gmail-m_828764688535476484gmail-m_4731746008415492739gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Tom Payerle <br>DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a><br></div><div>5825 University Research Park (301) 405-6135<br></div><div dir="ltr">University of Maryland<br>College Park, MD 20740-3831<br></div></div></div></div></div></div>
</blockquote></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_8508270196494718028gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Tom Payerle <br>DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a><br></div><div>5825 University Research Park (301) 405-6135<br></div><div dir="ltr">University of Maryland<br>College Park, MD 20740-3831<br></div></div></div></div></div></div>
</blockquote></div></div>