<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi David,<br>
<br>
if I remember right (we have disabled swap for years now), swapping
out processes seem to slow down the system overall.<br>
But I know, that if the oom_killer does its job (killing over memory
processes), the whole system is stalled until it has done its work.
This might be the issue, your users see.<br>
<br>
Hwloc at least should help the scheduler to decide, where to place
processes, but if I remember right, slurm has to be built with hwloc
support (meaning at least hwloc-devel has to be installed).<br>
But this part is more guessing, than knowing.<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="moz-cite-prefix">On 11/5/19 11:58 AM, David Baker wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CWXP265MB0376CA93F1CE5DE70A999126FE7E0@CWXP265MB0376.GBRP265.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Thank you for your replies. I double checked that the "task" in,
for example, taskplugin=task/affinity is optional. In this
respect it is good to know that we have the correct cgroups
setup. So in theory users should only disturb themselves,
however in reality we find that there is often a knock on effect
on other users' jobs. So, for example, users have complained
that their jobs sometimes stall. I can only vaguely think that
something odd is going on at the kernel level perhaps.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
One additional thing that I need to ask is... Should we have
hwloc installed our compute nodes? Does that help? Whenever I
check which processes are not being constrained by cgroups I
only ever find a small group of system processes.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on
behalf of Marcus Wagner <a class="moz-txt-link-rfc2396E" href="mailto:wagner@itc.rwth-aachen.de"><wagner@itc.rwth-aachen.de></a><br>
<b>Sent:</b> 05 November 2019 07:47<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] Running job using our serial
queue</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">Hi David,<br>
<br>
doing it the way you do it, is the same way, we do it.<br>
<br>
When the Matlab job asks for one CPU, it only gets on CPU this
way. That means, that all the processes are bound to this one
CPU. So (theoretically) the user is just disturbing himself, if
he uses more.<br>
<br>
But especially Matlab, there are more things to do. I t does not
suffice to add '-singleCompThread' to the commandline. Matlab is
not the only tool, that tries to use all cores, it finds on the
node.<br>
The same is valid for CPLEX and Gurobi, both often used from
Matlab. So even, if the user sets '-singleCompThread' for
Matlab, that does not mean at all, the job is only using one
CPU.<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
<div class="x_moz-cite-prefix">On 11/4/19 4:14 PM, David Baker
wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Hello,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
We decided to route all jobs requesting from 1 to 20 cores
to our serial queue. Furthermore, the nodes controlled by
the serial queue are shared by multiple users. We did this
to try to reduce the level of fragmentation across the
cluster -- our default "batch" queue provides exclusive
access to compute nodes. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
It looks like the downside of the serial queue is that jobs
from different users can interact quite badly. To some
extent this is an education issue -- for example matlab
users need to be told to add the "-singleCompThread" option
to their command line. On the other hand I wonder if our
cgroups setup is optimal for the serial queue. Our
<b>cgroup.conf</b> contains...</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<span><b>CgroupAutomount=yes<br>
</b></span>
<div><b>CgroupReleaseAgentDir="/etc/slurm/cgroup"<br>
</b></div>
<div><b><br>
</b></div>
<div><b>ConstrainCores=yes<br>
</b></div>
<div><b>ConstrainRAMSpace=yes<br>
</b></div>
<div><b>ConstrainDevices=yes<br>
</b></div>
<div><b>TaskAffinity=no<br>
</b></div>
<div><b><br>
</b></div>
<span><b>CgroupMountpoint=/sys/fs/cgroup</b></span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<span><br>
</span></div>
<div style=""><font face="Calibri, Arial, Helvetica,
sans-serif" color="#000000"><span style="font-size:12pt">The
</span>relevant<span style="font-size:12pt"> cgroup
configuration in the <b>slurm.conf
</b>is...</span></font></div>
<div style=""><b><span>ProctrackType=proctrack/cgroup<br>
</span><span>TaskPlugin=affinity,cgroup</span></b><br>
</div>
<div style=""><br>
</div>
<div style="">Could someone please advise us on the
required/recommended cgroup setup for the above scenario?
For example, should we really set "TaskAffinity=yes"? I
assume the interaction between jobs (sometimes jobs can get
stalled) is due to context switching at the kernel level,
however (apart from educating users) how can we minimise
that switching on the serial nodes?</div>
<div style=""><br>
</div>
<div style="">Best regards,</div>
<div style="">David</div>
<div style=""><br>
</div>
</blockquote>
<br>
<pre class="x_moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="x_moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de" moz-do-not-send="true">wagner@itc.rwth-aachen.de</a>
<a class="x_moz-txt-link-abbreviated" href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.itc.rwth-aachen.de&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cf4fb53d4fef74523599b08d761c4ac18%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=dtF928nvXUbXjpc4COy5bB9Qrs9LoZE8ePa26Ydjdsc%3D&reserved=0" originalsrc="http://www.itc.rwth-aachen.de" shash="x3eqMC26oq7pQ4TTGpY8KQ6Fn2PphDD8dmIWS99c3VpsyFlskv8PtUl7iFPrzc4MetSknFKcc0J79a7pUxWOY/c5oxn4Y3agSkUMcebz8dU2Q6kej6m+CmsZbZCnVmTuVKwNRRO8/RLcj+r3mN5mUo/oAqLBnobDTMKKO/iY4j0=" moz-do-not-send="true">www.itc.rwth-aachen.de</a>
</pre>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="moz-txt-link-abbreviated" href="http://www.itc.rwth-aachen.de">www.itc.rwth-aachen.de</a>
</pre>
</body>
</html>