<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0; margin-bottom:0">Hello Doug,</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Thank you for your detailed reply regarding how to setup backfill. There's quite a lot to take in there. Fortunately, I now have a day or two to read up and understand the ideas now that our cluster is down due to a
water cooling failure. In the first instance, I'll certainly implement bf_continue and review/amend the "bf_maxjobs" and "bf_interval" parameters. Switching on backfill debugging sounds very useful, but does that setting tend to blot the logs if left enabled
for long periods?</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">We did have a contract with SchedMD which recently finished. In one of the last discussions we had it was intimated that we may have hit a bug. That's in the respect that backfilled jobs were potentially stealing nodes
intended for higher priority jobs -- bug 5297. The advice was to consider upgrading to slurm 18.08.4 and implement
<b>bf_ignore_newly_avail_nodes</b>. I was interested to see that you had a similar discussion with SchedMD and did upgrade. I think I ought to update the bf configuration re my first paragraph and see how that goes before we bite the bullet and do the upgrade
(we are at 18.08.0 currently).</p>
<p style="margin-top:0; margin-bottom:0"><br>
</p>
<p style="margin-top:0; margin-bottom:0">Best regards,</p>
<p style="margin-top:0; margin-bottom:0">David</p>
<br>
<div style="color:rgb(0,0,0)">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Douglas Jacobsen <dmjacobsen@lbl.gov><br>
<b>Sent:</b> 23 March 2019 13:30<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Backfill advice</font>
<div> </div>
</div>
<div>
<div>
<div dir="auto">Hello,</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">At first blush bf_continue and bf_interval as well as bf_maxjobs (if I remembered the parameter correctly) are critical first steps in tuning. Setting DebugFlags=backfill is essential to getting the needed data to make tuning decisions.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Use of per user/account settings if they are too low can also cause starvation depending on the way your priority calculation is set up.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I presented these slides a few years ago ag the slurm user group on this topic:
<div><a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2FSLUG16%2FNERSC.pdf&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C479a97721a87443f81c708d6af9455dd%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=OC57jAeprk%2Bm1tCCn20mtAVzfvmbcj4AgJ5b3wVPJKI%3D&reserved=0" originalsrc="https://slurm.schedmd.com/SLUG16/NERSC.pdf" shash="Eh6IK6optxkEIYphuMWR7QeVLIvjGxBf11nSospkN4ULK5RJiZ+3/JNDU3pmCLVfwhMCshfXx0VMODrxditJo5jWD3imjHNwmUF2nxi7hgiJGaPr987eeTudwD8iO76y8ShqcRe9rDY8ps9AS7psN/q9pL087Gs0iXcLl7GbPHM=" id="LPlnk843996" class="OWAAutoLink" previewremoved="true">https://slurm.schedmd.com/SLUG16/NERSC.pdf</a></div>
<div dir="auto"><br>
</div>
<div dir="auto">The key thing to keep in mind with large jobs is that slurm needs to evaluate them again and again in the same order or the scheduled time may drift. Thus it is important that once jobs are getting planning reservations they must continue to
do so.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Because of the prevalence of large jobs at our site we use bf_min_prio_resv which splits the priority space into a reserving and non-reserving set, and then use job age to allow jobs to age from the non reserving portion of the priority space
to the reservation portion. Use of the recent MaxJobsAccruePerUser limits on a job qos can throttle the rate of jobs aging and prevent negative effects from users submitting large numbers of jobs.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I realize that is a large number of tunables and concepts densely packed, but it should give you some reasonable starting points.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Doug</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
<div>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Sat, Mar 23, 2019 at 05:26 david baker <<a href="mailto:djbaker12@gmail.com" id="LPlnk488237" class="OWAAutoLink" previewremoved="true">djbaker12@gmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>We do have large jobs getting starved out on our cluster, and I note particularly that we never manage to see a job getting assigned a start time. It seems very possible that backfilled jobs are stealing nodes reserved for large/higher priority jobs.<br>
</div>
<div><br>
</div>
<div>I'm wondering if our backfill configuration has any bearing on this issue or whether we are unfortunate enough to have hit a bug. One parameter that is missing in our bf setup is "bf_continue". Is that parameter significant in terms of ensuring that bf
drills down sufficiently in the job mix? Also we are using the default bf frequency -- should we really reduce the frequency and potentially reduce the number of bf jobs per group/user or total at each iteration? Currently, I think we are setting the per/user
limit to 20.</div>
<div><br>
</div>
<div>Any thoughts would be appreciated, please.</div>
<div><br>
</div>
<div>Best regards,</div>
<div>David</div>
</div>
</blockquote>
</div>
</div>
-- <br>
<div dir="ltr" class="x_gmail_signature">Sent from Gmail Mobile</div>
</div>
</div>
</div>
</body>
</html>