<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
You can check with something like this inside of a job: cat
/sys/fs/cgroup/cpuset/slurm/uid_$UID/job_$SLURM_JOB_ID/cpuset.cpus.
That lists which cpus you have access to.<br>
<br>
<div class="moz-cite-prefix">On 5/14/21 4:40 PM, Renfro, Michael
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN7PR07MB4241B3A8F39BE1DB8DDD3735A2509@BN7PR07MB4241.namprd07.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Untested, but prior experience with cgroups
indicates that if things are working correctly, even if your
code tries to run as many processes as you have cores, those
processes will be confined to the cores you reserve.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Try a more compute-intensive worker
function that will take some seconds or minutes to complete,
and watch the reserved node with 'top' or a similar program.
If for example, the job reserved only 1 core and tried to run
20 processes, you'd see 20 processes in 'top', each at 5% CPU
time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To make the code a bit more polite, you can
import the os module and create a new variable from the
SLURM_CPUS_ON_NODE environment variable to guide Python into
starting the correct number of processes:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> cpus_reserved =
int(os.environ['SLURM_CPUS_ON_NODE'])<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span
style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of
Rodrigo Santibáñez <a class="moz-txt-link-rfc2396E" href="mailto:rsantibanez.uchile@gmail.com"><rsantibanez.uchile@gmail.com></a><br>
<b>Date: </b>Friday, May 14, 2021 at 5:17 PM<br>
<b>To: </b>Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] Exposing only requested
CPUs to a job on a given node.<o:p></o:p></span></p>
</div>
<p style="margin:0in;text-align:center;background:white"
align="center"><b><span
style="font-size:12.0pt;color:red;background:white">External
Email Warning</span></b><o:p></o:p></p>
<p
style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white"
align="center">
<b><span style="font-size:12.0pt;color:red">This email
originated from outside the university. Please use caution
when opening attachments, clicking links, or responding to
requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" style="text-align:center" align="center">
<hr width="100%" size="0" align="center">
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hi you all,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm replying to have notifications
answering this question. I have a user whose python
script used almost all CPUs, but configured to use only
6 cpus per task. I reviewed the code, and it doesn't
have an explicit call to multiprocessing or similar. So
the user is unaware of this behavior (and also me).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Running slurm 20.02.6<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best!<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, May 14, 2021 at 1:37 PM Luis
R. Torres <<a href="mailto:lrtorres@gmail.com"
moz-do-not-send="true">lrtorres@gmail.com</a>>
wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC
1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Folks, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are currently running on SLURM
20.11.6 with cgroups constraints for memory and
CPU/Core. Can the scheduler only expose the
requested number of CPU/Core resources to a job? We
have some users that employ python scripts with the
multi processing modules, and the scripts apparently
use all of the CPU/Cores in a node, despite using
options to constraint a task to just a given number
of CPUs. We would like several multiprocessing
jobs to run simultaneously on the nodes, but not
step on each other.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The sample script I use for
testing is below; I'm looking for something similar
to what can be done with the GPU Gres configuration
where only the number of GPUs requested are exposed
to the job requesting them. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">#!/usr/bin/env
python3<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">import
multiprocessing<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;min-height:21px"><span
style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">def
worker():<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">
print("Worker on CPU #%s" %
multiprocessing.current_process<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">().name)<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> result=0<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> for j in
range(20):<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> result
+= j**2<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> print
("Result on CPU {} is
{}".format(multiprocessing.curr<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">ent_process().name,result))<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> return <o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;min-height:21px"><span
style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">if __name__
== '__main__':<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> pool =
multiprocessing.Pool()<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> jobs =
[]<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> print
("This host exposed {}
CPUs".format(multiprocessing.c<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">pu_count()))<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> for i in
range(multiprocessing.cpu_count()):<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black"> p =
multiprocessing.Process(target=worker,
name=i).star<o:p></o:p></span></p>
<p
style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal"><span
style="font-family:Menlo;color:black">t()<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">----------------------------------------<br>
Luis R. Torres<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Ryan Cox
Director
Office of Research Computing
Brigham Young University
</pre>
</body>
</html>