<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I currently manage a small cluster separated into 4 partitions. I am experiencing unexpected behavior with the scheduler when the queue has been flooded with a large number of jobs by a single user (around
60000) to a single partition. We have each user bound to a global grptres CPU limit. Once this user reaches their CPU limit the jobs are queued with reason “AssocGroupCpuLimit” but after a few hundred or so of the jobs it seems to switch to “Priority”. The
issue is that once this switch occurs it appears to also impact all other partitions. Currently if any job is submitted to any of the partitions, regardless of resources available, they are all queued by the scheduler with the reason of “Priority”. We had
the scheduler initially configured for backfill but have also tried switching to builtin and it did not seem to make a difference. I tried increasing the default_queue_depth to 100000 and it didn’t seem to help. The scheduler log is also unhelpful as it simply
lists the accounting-limited jobs and never mentions the “Priority” queued jobs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:53.993] JobId=495780 delayed for accounting policy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:53.997] JobId=495781 delayed for accounting policy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.001] JobId=495782 delayed for accounting policy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.005] JobId=495783 delayed for accounting policy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sched: [2021-06-11T13:21:54.005] loop taking too long, breaking out<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’ve gone through all the documentation I’ve found on the scheduler and cannot seem to resolve this. I’m hoping I’m simply missing something.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any help would be great. Thank you!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:black">Jason </span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>