<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Michael,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style=""><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif"><span style="font-size: 12pt;">Thank you for your reply and your thoughts. These are the priority weights that I have configured in the slurm.conf. </span></font></div>
<div style=""><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif"><span style="font-size: 12pt;"><br>
</span></font></div>
<div style=""><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif"><span style="font-size: 12pt;"><span>PriorityWeightFairshare=1000000<br>
</span>
<div>PriorityWeightAge=100000<br>
</div>
<div>PriorityWeightPartition=1000</div>
<div>PriorityWeightJobSize=10000000<br>
</div>
<span>PriorityWeightQOS=10000</span><br>
</span></font></div>
<div style=""><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif"><span style="font-size: 12pt;"><br>
</span></font></div>
<div style=""><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif"><span style="font-size: 12pt;">I've made the PWJobSize to be the highest factor, however I understand that that only provides a once-off kick to jobs and so it probably
</span>insignificant in the longer run<span style="font-size: 12pt;"> . That's followed by the PWFairshare. </span></font></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Should I really be looking at increasing the PWAge factor to help to "push jobs" through the system? </span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>The other issue that might play a part is that we see a lot of single node jobs (presumably backfilled) into the system. Users aren't excessively bombing the cluster, but maybe some backfill throttling would be useful as well (?)</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>What are your thoughts having seen the priority factors, please? I've attached a copy of the slurm.conf just in case you or anyone else wants to take a more complete overview.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Gutteridge <michael.gutteridge@gmail.com><br>
<b>Sent:</b> 09 April 2019 18:59<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Effect of PriorityMaxAge on job throughput</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="x_gmail_default" style="font-family:monospace"><br>
</div>
<div class="x_gmail_default" style="font-family:monospace">It might be useful to include the various priority factors you've got configured. The fact that adjusting PriorityMaxAge had a dramatic effect suggests that the age factor is pretty high- might be
worth looking at that value relative to the other factors.</div>
<div class="x_gmail_default" style="font-family:monospace"><br>
</div>
<div class="x_gmail_default" style="font-family:monospace">Have you looked at PriorityWeightJobSize? Might have some utility if you're finding large jobs getting short-shrift.</div>
<div class="x_gmail_default" style="font-family:monospace"><br>
</div>
<div class="x_gmail_default" style="font-family:monospace"> - Michael</div>
<div class="x_gmail_default" style="font-family:monospace"><br>
</div>
<input name="x_virtru-metadata" type="hidden"></div>
</div>
<br>
<div class="x_gmail_quote" style="">
<div dir="ltr" class="x_gmail_attr">On Tue, Apr 9, 2019 at 2:01 AM David Baker <<a href="mailto:D.J.Baker@soton.ac.uk">D.J.Baker@soton.ac.uk</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I've finally got the job throughput/turnaround to be reasonable in our cluster. Most of the time the job activity on the cluster sets the default QOS to 32 nodes (there are 464 nodes in the default queue). Jobs requesting nodes close to the QOS level (for example
22 nodes) are scheduled within 24 hours which is better than it has been. Still I suspect there is room for improvement. I note that these large jobs still struggle to be given a starttime, however many jobs are now being given a starttime following my SchedulerParameters
makeover.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I used advice from the mailing list and the Slurm high throughput document to help me make changes to the scheduling parameters. They are now...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
SchedulerParameters=assoc_limit_continue,batch_sched_delay=20,bf_continue,bf_interval=300,bf_min_age_reserve=10800,bf_window=3600,bf_resolution=600,bf_yield_interval=1000000,partition_job_depth=500,sched_max_job_start=200,sched_min_interval=2000000<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Also..</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>PriorityFavorSmall=NO<br>
</span>
<div>PriorityFlags=SMALL_RELATIVE_TO_TIME,ACCRUE_ALWAYS,FAIR_TREE<br>
</div>
<div>PriorityType=priority/multifactor<br>
</div>
<span>PriorityDecayHalfLife=7-0</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>PriorityMaxAge=1-0<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>The most significant change was actually reducing "PriorityMaxAge" to 7-0 to 1-0. Before that change the larger jobs could hang around in the queue for days. Does it make sense therefore to further reduce PriorityMaxAge to less than 1 day? Your advice
would be appreciated, please.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>Best regards,</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>David</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>