<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>We've seen this in our shop. Our solutions have been:</p>
<p>1. User defer or max_rpc_cnt to slow down the scheduler so it can
catch up with RPC's</p>
<p>2. Target specific chatty users and tell them to knock it off.
sdiag is your friend for this. We also repeatedly tell users not
to ping the scheduler more often than once a minute.<br>
</p>
<p>3. We built a caching DB for squeue that gives users almost live
data instead of live data. So when they hit squeue that hit an
external database rather than the scheduler itself.</p>
<p>Also making sure you are on the latest version of Slurm is highly
recommended as there are numerous performance improvements.</p>
<p>For something straight out of the box though I would look at
defer/max_rpc_cnt as that will help the scheduler cope with high
RPC traffic.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 8/17/2020 2:30 PM, Ransom, Geoffrey
M. wrote:<br>
</div>
<blockquote type="cite"
cite="mid:39dc7adcdcb44d87a9ca66bbd7a6b38e@APLEX03.dom1.jhuapl.edu">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello<o:p></o:p></p>
<p class="MsoNormal"> We are having performance issues with
slurtmctld (delayed sinfo/squeue results, socket timeouts for
multiple sbatch calls, jobs/nodes sitting in COMP state for an
extended period of time).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We just fully switch to Slurm from Univa
and I think our problem is users putting a lot of “scontrol
show” calls (maybe squeue/sinfo as well) in large batches of
jobs and essentially DOS-ing our scheduler.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there a built in way to throttle
“squeue/sinfo/scontrol show” commands in a reasonable manner
so one user can’t do something dumb running jobs that keep
calling these commands in bulk?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If I need to make something up to verify
this I am thinking about making a wrapper script around these
commands that locks a shared temp file on the local disk (to
avoid NFS locking issues) of each machine and then sleeps for
5 seconds before calling the real command and releasing the
lock. At least this way a user will only run 1 copy per
machine that their jobs land on instead of 1 per CPU core per
machine.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks.<o:p></o:p></p>
</div>
</blockquote>
</body>
</html>