<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Thank you for your reply and for the explanation. That makes sense -- your explanation of backfill is as we expected. I think it's more that we are surprised that almost all our jobs were being scheduled using backfill.
We very rarely see any being scheduled normally. It could be that we haven't actually tuned our priority weights particularly well. We potentially need a setup that will allow users to everything from small (including very small, small duration, test jobs
with a high QOS) to large jobs running over a range of times without too many users losing out. Initially, we had our Age and Job size scaling factors too low, but have currently got the setup shown below. </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Any thoughts, please? </p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0">Best regards,</p>
<p style="margin-top:0;margin-bottom:0">David</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>PriorityParameters = (null)</div>
<div>PriorityDecayHalfLife = 14-00:00:00</div>
<div>PriorityCalcPeriod = 00:05:00</div>
<div>PriorityFavorSmall = No</div>
<div>PriorityFlags = SMALL_RELATIVE_TO_TIME,DEPTH_OBLIVIOUS</div>
<div>PriorityMaxAge = 14-00:00:00</div>
<div>PriorityUsageResetPeriod = NONE</div>
<div>PriorityType = priority/multifactor</div>
<div>PriorityWeightAge = 100000</div>
<div>PriorityWeightFairShare = 100000</div>
<div>PriorityWeightJobSize = 100000</div>
<div>PriorityWeightPartition = 0</div>
<div>PriorityWeightQOS = 1000000</div>
<div>PriorityWeightTRES = (null)</div>
<div>PropagatePrioProcess = 0</div>
<p></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Loris Bennett <loris.bennett@fu-berlin.de><br>
<b>Sent:</b> 20 November 2018 13:26:14<br>
<b>To:</b> Baker D.J.<br>
<b>Cc:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Excessive use of backfill on a cluster</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi David,<br>
<br>
Baker D.J. <D.J.Baker@soton.ac.uk> writes:<br>
<br>
> Hello,<br>
><br>
> We are running Slurm 18.08.0 on our cluster and I am concerned that<br>
> Slurm appears to be using backfill scheduling excessively. In fact the<br>
> vast majority of jobs are being scheduled using backfill. So, for<br>
> example, I have just submitted a set of three serial jobs. They all<br>
> started on a compute node that was completely free, but<br>
> disconcertingly in the slurmctl log they were all reported as started<br>
> using backfill and that isn't making sense...<br>
><br>
> [2018-11-20T12:31:27.598] backfill: Started JobId=217031 in batch on red158<br>
> [2018-11-20T12:32:28.004] backfill: Started JobId=217032 in batch on red158<br>
> [2018-11-20T12:33:58.608] backfill: Started JobId=217033 in batch on red158<br>
><br>
> I either don't understand the context of backfill re slurm or the<br>
> above is odd. Has anyone seem this "overuse" (unnecessary) use of<br>
> backfill on their cluster and/or could offer advice, please.<br>
<br>
I am not sure what "excessive backfilling" might mean. If you have<br>
a job which requires a large amount of resources to become available<br>
before it can start, then backfilling will allow other jobs with a lower<br>
priority to be run, if this can be achieved without delaying the start<br>
of the large job. So if a job needs 100 nodes, at some point 99 of them<br>
will be idle. Job which can start and finish before the 100th node<br>
becomes available will indeed be backfilled on empty nodes. This is how<br>
backfilling is supposed to work.<br>
<br>
Or am I misunderstanding your problem?<br>
<br>
Cheers,<br>
<br>
Loris<br>
<br>
-- <br>
Dr. Loris Bennett (Mr.)<br>
ZEDAT, Freie Universität Berlin Email loris.bennett@fu-berlin.de<br>
</div>
</span></font></div>
</body>
</html>