<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0; margin-bottom:0">Hi Cyrus,</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Thank you for the links. I've taken a good look through the first link (re the cloud cluster) and the only parameter that might be relevant is "assoc_limit_stop", but I'm not sure if that is relevant in this instance.
The reason for the delay of the job in question is "priority", however there are quite a lot of jobs from users in the same accounting group with jobs delayed due to "<span>QOSMaxCpuPerUserLimit". They also talk about using the "builtin" scheduler which I
guess would turn off backfill.</span></p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">I have attached a copy of the current slurm.conf so that you and other members can get a better feel for the whole picture. Certainly we see a large number of serial/small (1 node) jobs running through the system and
I'm concerned that my setup encourages this behaviour, however how to stem this issue is a mystery to me.</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">If you or anyone else has any relevant thoughts then please let me know. I particular I am keen to understand "assoc_limit_stop" and whether it is a relevant option in this situation.</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Best regards,</p>
<p style="margin-top:0; margin-bottom:0">David</p>
<br>
<div style="color:rgb(0,0,0)">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Cyrus Proctor <cproctor@tacc.utexas.edu><br>
<b>Sent:</b> 21 March 2019 14:19<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> Re: [slurm-users] Very large job getting starved out</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">
<p>Hi David,</p>
<p><br>
</p>
<p>You might have a look at the thread "Large job starvation on cloud cluster" that started on Feb 27; there's some good tidbits in there. Off the top without more information, I would venture that settings you have in slurm.conf end up backfilling the smaller
jobs at the expense of scheduling the larger jobs.</p>
<p><br>
</p>
<p>Your partition configs plus accounting and scheduler configs from slurm.conf would be helpful.</p>
<p><br>
</p>
<p>Also, search for "job starvation" here: <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fsched_config.html&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cea23798d0ad54a02f14308d6ae0883d5%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=KfjAqNHQgLcUBBYwZFi8OygU2De%2FdVuTwbdOmUv0Dps%3D&reserved=0" originalsrc="https://slurm.schedmd.com/sched_config.html" shash="w4zujlmkpPgq4opZEIY6YLYwBYa+5k4v9O3JQrA7hQkX+Yys09awvZDggR95mxxsnK+NpVyY+UfLQDaT8y014JGG70bBQkvd7Tu2EMEJleP/O/q6q1tbodkzlcwSVXjqQxSqhP1+oD/EvtFyBG3dI3OSonGiZalKa8c3Tb2/aw0=" id="LPlnk295357" class="OWAAutoLink" previewremoved="true">
https://slurm.schedmd.com/sched_config.html</a> as another potential starting point.</p>
<p><br>
</p>
<p>Best,</p>
<p>Cyrus</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 3/21/19 8:55 AM, David Baker wrote:<br>
</div>
<blockquote type="cite">
<div id="x_divtagdefaultwrapper" dir="ltr" style="">
<p style="margin-top:0; margin-bottom:0">Hello,</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">I understand that this is not a straight forward question, however I'm wondering if anyone has any useful ideas, please. Our cluster is busy and the QOS has limited users to a maximum of 32 compute nodes on the "batch"
queue. Users are making good of the cluster -- for example one user is running five 6 node jobs at the moment. On the other hand, a job belonging to another user has been stalled in the queue for around 7 days. He has made reasonable use of the cluster and
as a result his fairshare component is relatively low. Having said that, the priority of his job is high -- it currently one of the highest priority jobs in the batch partition queue. From sprio...</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0"><span>JOBID PARTITION PRIORITY AGE FAIRSHARE JOBSIZE PARTITION QOS</span><br>
</p>
<p style="margin-top:0; margin-bottom:0"><span>359323 batch 180292 100000 79646 547 100 0</span><br>
</p>
<p style="margin-top:0; margin-bottom:0"><span><br>
</span></p>
<p style="margin-top:0; margin-bottom:0"><span>I did think that the PriorityDecayHalfLife was quite high at 14 days and so I reduced that to 7 days. For reference I've included the key scheduling settings from the cluster below. Does anyone have any thoughts,
please? </span></p>
<p style="margin-top:0; margin-bottom:0"><span><br>
</span></p>
<p style="margin-top:0; margin-bottom:0"><span>Best regards,</span></p>
<p style="margin-top:0; margin-bottom:0"><span>David</span></p>
<p style="margin-top:0; margin-bottom:0"><span><br>
</span></p>
<p style="margin-top:0; margin-bottom:0"><span></span></p>
<div>PriorityDecayHalfLife = 7-00:00:00</div>
<div>PriorityCalcPeriod = 00:05:00</div>
<div>PriorityFavorSmall = No</div>
<div>PriorityFlags = ACCRUE_ALWAYS,SMALL_RELATIVE_TO_TIME,FAIR_TREE</div>
<div>PriorityMaxAge = 7-00:00:00</div>
<div>PriorityUsageResetPeriod = NONE</div>
<div>PriorityType = priority/multifactor</div>
<div>PriorityWeightAge = 100000</div>
<div>PriorityWeightFairShare = 1000000</div>
<div>PriorityWeightJobSize = 10000000</div>
<div>PriorityWeightPartition = 1000</div>
<div>PriorityWeightQOS = 10000</div>
<br>
<p style="margin-top:0; margin-bottom:0"><span><br>
</span></p>
<p style="margin-top:0; margin-bottom:0"><span><br>
</span></p>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>