<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hi Lois</p>
<br>
Thank you for sharing your multi priority configuration with us. I understand why you say about the QOS factor -- I've reduced it and increased the FS factor to see where that takes us. Our QOS factor is only there to ensure that test jobs gain a higher priority
more quickly than other jobs, however on reflection our high QOS factor setting was well over the top. </div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
We turned on a higher debugging level this morning to help us understand the situation better. There is a definite split between small and larger jobs. Given that our user group is growing and we only have 750 compute nodes then perhaps we should expect to
see a lot of backfill while resources become available for larger jobs. </div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
Best regards,</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
David</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;" dir="ltr">
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Loris Bennett <loris.bennett@fu-berlin.de><br>
<b>Sent:</b> 20 November 2018 15:58<br>
<b>To:</b> Baker D.J.<br>
<b>Cc:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Excessive use of backfill on a cluster</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">Hi David,<br>
<br>
We have<br>
<br>
PriorityType=priority/multifactor<br>
PriorityDecayHalfLife=14-0<br>
PriorityWeightFairshare=10000000<br>
PriorityWeightAge=10000<br>
PriorityWeightPartition=10000<br>
PriorityWeightJobSize=0<br>
PriorityWeightQOS=10000<br>
PriorityMaxAge=7-0<br>
PriorityCalcPeriod=5<br>
SchedulerType=sched/backfill<br>
SchedulerParameters=max_job_bf=50,bf_interval=60,bf_window=20160,default_queue_depth=1000<br>
<br>
In particular our main priority factor, by a long way, is Fairshare,<br>
with a slight advantage for old jobs and QOSs with a short run-time.<br>
With your priority weights, QOS is the most important by a factor of 10.<br>
I'm not quite sure what effects this will have, other than that your<br>
priorities will be a bit more static, since the total priority will have<br>
a reduced time-dependency.<br>
<br>
We had to add the SchedulerParameters settings to get backfill working<br>
properly at all, but this obviously isn't your problem.<br>
<br>
Cheers,<br>
<br>
Loris<br>
<br>
Baker D.J. <D.J.Baker@soton.ac.uk> writes:<br>
<br>
> Hello,<br>
><br>
> Thank you for your reply and for the explanation. That makes sense --<br>
> your explanation of backfill is as we expected. I think it's more that<br>
> we are surprised that almost all our jobs were being scheduled using<br>
> backfill. We very rarely see any being scheduled normally. It could be<br>
> that we haven't actually tuned our priority weights particularly<br>
> well. We potentially need a setup that will allow users to everything<br>
> from small (including very small, small duration, test jobs with a<br>
> high QOS) to large jobs running over a range of times without too many<br>
> users losing out. Initially, we had our Age and Job size scaling<br>
> factors too low, but have currently got the setup shown below.<br>
><br>
> Any thoughts, please? <br>
><br>
> Best regards,<br>
><br>
> David<br>
><br>
> PriorityParameters = (null)<br>
> PriorityDecayHalfLife = 14-00:00:00<br>
> PriorityCalcPeriod = 00:05:00<br>
> PriorityFavorSmall = No<br>
> PriorityFlags = SMALL_RELATIVE_TO_TIME,DEPTH_OBLIVIOUS<br>
> PriorityMaxAge = 14-00:00:00<br>
> PriorityUsageResetPeriod = NONE<br>
> PriorityType = priority/multifactor<br>
> PriorityWeightAge = 100000<br>
> PriorityWeightFairShare = 100000<br>
> PriorityWeightJobSize = 100000<br>
> PriorityWeightPartition = 0<br>
> PriorityWeightQOS = 1000000<br>
> PriorityWeightTRES = (null)<br>
> PropagatePrioProcess = 0<br>
><br>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
> From: Loris Bennett <loris.bennett@fu-berlin.de><br>
> Sent: 20 November 2018 13:26:14<br>
> To: Baker D.J.<br>
> Cc: Slurm User Community List<br>
> Subject: Re: [slurm-users] Excessive use of backfill on a cluster <br>
> Hi David,<br>
><br>
> Baker D.J. <D.J.Baker@soton.ac.uk> writes:<br>
><br>
>> Hello,<br>
>><br>
>> We are running Slurm 18.08.0 on our cluster and I am concerned that<br>
>> Slurm appears to be using backfill scheduling excessively. In fact the<br>
>> vast majority of jobs are being scheduled using backfill. So, for<br>
>> example, I have just submitted a set of three serial jobs. They all<br>
>> started on a compute node that was completely free, but<br>
>> disconcertingly in the slurmctl log they were all reported as started<br>
>> using backfill and that isn't making sense...<br>
>><br>
>> [2018-11-20T12:31:27.598] backfill: Started JobId=217031 in batch on red158<br>
>> [2018-11-20T12:32:28.004] backfill: Started JobId=217032 in batch on red158<br>
>> [2018-11-20T12:33:58.608] backfill: Started JobId=217033 in batch on red158<br>
>><br>
>> I either don't understand the context of backfill re slurm or the<br>
>> above is odd. Has anyone seem this "overuse" (unnecessary) use of<br>
>> backfill on their cluster and/or could offer advice, please.<br>
><br>
> I am not sure what "excessive backfilling" might mean. If you have<br>
> a job which requires a large amount of resources to become available<br>
> before it can start, then backfilling will allow other jobs with a lower<br>
> priority to be run, if this can be achieved without delaying the start<br>
> of the large job. So if a job needs 100 nodes, at some point 99 of them<br>
> will be idle. Job which can start and finish before the 100th node<br>
> becomes available will indeed be backfilled on empty nodes. This is how<br>
> backfilling is supposed to work.<br>
><br>
> Or am I misunderstanding your problem?<br>
><br>
> Cheers,<br>
><br>
> Loris<br>
<br>
-- <br>
Dr. Loris Bennett (Mr.)<br>
ZEDAT, Freie Universität Berlin Email loris.bennett@fu-berlin.de<br>
</div>
</span></font></div>
</div>
</div>
</div>
</body>
</html>