<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you for the response!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I have given those parameters a shot and will monitor the queue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">These parameters would really only impact backfill with respect to job time limits, correct? Based on what I have read, I was under the impression that the main scheduler and the backfill scheduler were partition
independent. Meaning that if I have a large number of jobs queued for a single partition, another partition which has no jobs running would be seen as free by the scheduler and should not queue jobs with respect to the filled partition. Do you know if I’m
mistaken?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Jason Dana<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">JHUAPL<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">REDD/RA2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Senior Systems Administrator/Software Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"><a href="mailto:jason.dana@jhuapl.edu"><span style="color:#954F72">jason.dana@jhuapl.edu</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">240-564-1045 (w)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Need Support from </span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:red">REDD</span></b><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">? </span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">You
can enter a ticket using the new REDD Help Desk Portal (<a href="https://help.rcs.jhuapl.edu/servicedesk"><span style="color:#0563C1">https://help.rcs.jhuapl.edu/servicedesk</span></a>) if you have an active account or e-mail <a href="mailto:redd-help@outermail.jhuapl.edu"><span style="color:#954F72">redd-help@outermail.jhuapl.edu</span></a>.</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="color:black">From: </span>
</b><span style="color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "Renfro, Michael" <Renfro@tntech.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Friday, June 11, 2021 at 2:16 PM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com>, "slurm-users@schedmd.com" <slurm-users@schedmd.com><br>
<b>Subject: </b>[EXT] Re: [slurm-users] Slurm Scheduler Help<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<div id="APLWarningText">
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="100%" style="width:100.0%;background:#E0E0E0;padding:0in 0in 0in 0in">
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<b><span style="font-size:11.0pt;color:red">APL external email warning: </span></b><span style="font-size:11.0pt;color:black">Verify sender slurm-users-bounces@lists.schedmd.com before clicking links or attachments</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p style="margin-left:.5in"><span style="color:white"> </span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Not sure it would work out to 60k queued jobs, but we're using:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:.5in"><span style="font-size:11.0pt">SchedulerParameters=bf_window=43200,bf_resolution=2160,bf_max_job_user=80,bf_continue,default_queue_depth=200</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">in our setup. bf_window is driven by our 30-day max job time, bf_resolution is at 5% of that time, and the other values are just what we landed on. This did manage to address some
backfill issues we had in previous years.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<b><span style="color:black">From: </span></b><span style="color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Dana, Jason T. <Jason.Dana@jhuapl.edu><br>
<b>Date: </b>Friday, June 11, 2021 at 12:27 PM<br>
<b>To: </b>slurm-users@schedmd.com <slurm-users@schedmd.com><br>
<b>Subject: </b>[slurm-users] Slurm Scheduler Help</span><o:p></o:p></p>
</div>
<p align="center" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:48.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="margin-left:.5in;text-align:center">
<span style="font-size:11.0pt">
<hr size="0" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">I currently manage a small cluster separated into 4 partitions. I am experiencing unexpected behavior with the scheduler when the queue has been flooded with a large number of jobs
by a single user (around 60000) to a single partition. We have each user bound to a global grptres CPU limit. Once this user reaches their CPU limit the jobs are queued with reason “AssocGroupCpuLimit” but after a few hundred or so of the jobs it seems to
switch to “Priority”. The issue is that once this switch occurs it appears to also impact all other partitions. Currently if any job is submitted to any of the partitions, regardless of resources available, they are all queued by the scheduler with the reason
of “Priority”. We had the scheduler initially configured for backfill but have also tried switching to builtin and it did not seem to make a difference. I tried increasing the default_queue_depth to 100000 and it didn’t seem to help. The scheduler log is also
unhelpful as it simply lists the accounting-limited jobs and never mentions the “Priority” queued jobs:</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:53.993] JobId=495780 delayed for accounting policy</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:53.997] JobId=495781 delayed for accounting policy</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.001] JobId=495782 delayed for accounting policy</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.005] JobId=495783 delayed for accounting policy</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.005] loop taking too long, breaking out</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">I’ve gone through all the documentation I’ve found on the scheduler and cannot seem to resolve this. I’m hoping I’m simply missing something.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt">Any help would be great. Thank you!</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;color:black">Jason
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"> <o:p></o:p></p>
</div>
</div>
</body>
</html>