<div dir="ltr"><div>Hi Sean,</div><div>thank you very much for your reply.</div><div><br></div><div>> If a lower priority job can start AND finish before the resources a
higher priority job requires are available, the backfill scheduler will
start the lower priority job.</div><div><br></div><div>That's very interesting, but how can the scheduler predict how long a low-priority job will take? <br></div><div><div><br>> In your example job list, can you also list the requested times for each
job? That will show if it is the backfill scheduler doing what it is
designed to do.</div><div><br></div><div>You mean the wall clock time? If that's the case, we don't usually set that.</div><div><br></div><div>Thanks again</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 7, 2020 at 11:39 AM Sean Crosby <<a href="mailto:scrosby@unimelb.edu.au">scrosby@unimelb.edu.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>What you have described is how the backfill scheduler works. If a lower priority job can start AND finish before the resources a higher priority job requires are available, the backfill scheduler will start the lower priority job.</div><div><br></div><div>Your high priority job requires 24 cores, whereas the lower priority jobs only require 1 core each. Therefore there might be some free resources the lower priority jobs can use that the 24 core job can't. The backfill scheduler can make the lower priority jobs take advantage of those free cores, but only if it then doesn't stop the higher priority job from starting in its original time.<br></div><div><br></div><div>In your example job list, can you also list the requested times for each job? That will show if it is the backfill scheduler doing what it is designed to do.</div><div><br></div><div>Sean</div><div><br></div><div><div><div dir="ltr">--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 7 Jul 2020 at 19:05, zaxs84 <<a href="mailto:sciuscianebbia@gmail.com" target="_blank">sciuscianebbia@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif"><strong>UoM notice: External email. Be cautious of links, attachments, or impersonation attempts.</strong><br></div><hr></div>Hi all.<br><br>We want to achieve a simple thing with slurm: launch "normal" jobs, and be able to launch "high priority" jobs that run as soon as possible. End of it. However we cannot achieve this in a reliable way, meaning that our current config sometimes works, sometimes not, and this is driving us crazy.<br><br>When it works, this is what happens:<br>- we have, let's say, 10 jobs running with normal priority (--qos=normal, having final Priority=1001) and few thousands in PENDING state<br>- we submit a new job with high priority (--qos=high, having final Priority=1001001)<br>- at this point, slurm waits until the normal priority job will end to free up required resources, and then starts a new High priority job. That's Perfect!<br><br>However, from time to time, randomly, this does not happen. Here is an example:<br><br># the node has around 200GB of memory and 24 CPUs<br>Partition=t1 State=PD Priority=1001001 Nice=0 ID=337455 CPU=24 Memory=80G Nice=0 Started=0:00 User=u1 Submitted=2020-07-07T07:16:47<br>Partition=t1 State=R Priority=1001 Nice=0 ID=337475 CPU=1 Memory=1024M Nice=0 Started=1:22 User=u1 Submitted=2020-07-07T10:31:46<br>Partition=t1 State=R Priority=1001 Nice=0 ID=334355 CPU=1 Memory=1024M Nice=0 Started=58:09 User=u1 Submitted=2020-06-23T09:57:11<br>Partition=t1 State=R Priority=1001 Nice=0 ID=334354 CPU=1 Memory=1024M Nice=0 Started=6:29:59 User=u1 Submitted=2020-06-23T09:57:11<br>Partition=t1 State=R Priority=1001 Nice=0 ID=334353 CPU=1 Memory=1024M Nice=0 Started=13:25:55 User=u1 Submitted=2020-06-23T09:57:11<br>[...]<br><br>You see? Slurm keep starting jobs that have a lower priority. Why is that?<br><br>Some info about our config: Slurm is version 16.05. Here is the priority config of slurm:<br><br>##### file /etc/slurm-llnl/slurm.conf<br>PriorityType=priority/multifactor<br>PriorityFavorSmall=NO<br>PriorityWeightQOS=1000000<br>PriorityWeightFairshare=1000<br>PriorityWeightPartition=1000<br>PriorityWeightJobSize=0<br>PriorityWeightAge=0<br><br>##### command "sacctmgr show qos"<br> Name Priority MaxSubmitPA<br> normal 0 30<br> high 1000 <br><br><br>Any idea? <br><br>Thanks<br></div>
</blockquote></div>
</blockquote></div>