<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I've used Paul's `MaxJobs` suggestion in emergencies with success. +1 vote.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
We've encountered RPC timeouts and have been able to tune the `sched_max_job_start` (decrease) and `sched_min_interval` (increase) options of `SchedulerParameters` to reduce/eliminate timeouts during high job flux. Selecting good values required some trial
and error.<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Good luck!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Sebastian<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif;">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">
<div id="emailSignature" style="display:block">--</div>
<div id="emailSignature" style="display:block"><br>
<table style="box-sizing: border-box; border-collapse: collapse; border-spacing: 0px; background-color: rgb(255, 255, 255); max-width: 100%; text-align: start; font-family: Helvetica, Arial, san-serif; width: 480px;">
<tbody style="box-sizing:border-box">
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box; vertical-align:top" width="75"><a href="http://www.unr.edu/" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; color: rgb(46, 108, 162); text-decoration: underline;"><img alt="University of Nevada, Reno" style="box-sizing:border-box; vertical-align:middle" width="75" src="https://www.unr.edu/assets/images/blockn-100x100.png"></a></td>
<td style="box-sizing:border-box" width="20"> </td>
<td style="box-sizing:border-box; vertical-align:top" width="385">
<table style="box-sizing:border-box; border-collapse:collapse; max-width:100%; width:385px; text-align:left">
<tbody style="box-sizing:border-box">
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box; vertical-align:top"><span style="box-sizing:border-box"><span style="box-sizing: border-box; color: rgb(0, 51, 102);"><strong style="box-sizing:border-box; font-weight:bold"><span style="box-sizing:border-box">Sebastian Smith<br style="box-sizing:border-box">
</span></strong></span><span style="box-sizing:border-box"><span style="box-sizing:border-box">High-Performance Computing Engineer<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">Office of Information Technology<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">1664 North Virginia Street<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">MS 0291<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box"></span><span style="box-sizing:border-box"></span><br style="box-sizing:border-box">
<strong style="box-sizing:border-box; font-weight:bold">work-phone:<span> </span></strong><span style="box-sizing:border-box"><a href="tel:7756825050" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">775-682-5050</a><br style="box-sizing:border-box">
</span><strong style="box-sizing:border-box; font-weight:bold"></strong><span style="box-sizing:border-box"></span><strong style="box-sizing:border-box; font-weight:bold">email:<span> </span></strong><span style="box-sizing:border-box"><a href="mailto:stsmith@unr.edu" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">stsmith@unr.edu</a><br style="box-sizing:border-box">
</span><strong style="box-sizing:border-box; font-weight:bold">website:<span> </span></strong><span style="box-sizing:border-box"><a href="http://rc.unr.edu/" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">http://rc.unr.edu</a><br style="box-sizing:border-box">
</span></span><br style="box-sizing:border-box">
</span></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Paul Edmon <pedmon@cfa.harvard.edu><br>
<b>Sent:</b> Tuesday, September 22, 2020 5:01 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Quickly throttling/limiting a specific user's jobs</font>
<div> </div>
</div>
<div>
<p>I would look at:</p>
<dl compact="compact"><dt><i>MaxJobs</i>=<max jobs> </dt><dd>Maximum number of jobs each user is allowed to run at one time in this association. This is overridden if set directly on a user. Default is the cluster's limit. To clear a previously set value use the modify command with a new value of -1.
</dd></dl>
<p>Which is Association based. So you could just modify their account directly and set it to something low.</p>
<p>You can also simply put their pending jobs in hold state. That way they won't be considered for scheduling but won't be outright removed. Setting fairshare to 0 has the same effect.</p>
<p>-Paul Edmon-<br>
</p>
<div class="x_moz-cite-prefix">On 9/22/2020 7:58 PM, Brian Andrus wrote:<br>
</div>
<blockquote type="cite">
<p>Well, I know of no way to 'throttle' running jobs. Once they are out the gate, you can't stop them from leaving..</p>
<p>That said, your approach of setting arraytaskthrottle is just what you want for any pending jobs.</p>
<p>As a preventative measure, I imagine you could set the default to 1 and then change it with a job_submit script.</p>
<p>As far as currently running tasks, well, you have to figure that. You could kill/requeue them, but that can break things for the user. If their code supports it, they could checkpoint/restart as part of the process.</p>
<p>You can suspend them, but they still sit on the node waiting to be resumed, but the node resources may get assigned to other jobs while they wait to resume.</p>
<p>Brian Andrus<br>
</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 9/22/2020 2:33 PM, Ransom, Geoffrey M. wrote:<br>
</div>
<blockquote type="cite">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style></blockquote>
</blockquote>
</div>
</body>
</html>