<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Untested, but prior experience with cgroups indicates that if things are working correctly, even if your code tries to run as many processes as you have cores, those processes will be confined to the cores you reserve.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Try a more compute-intensive worker function that will take some seconds or minutes to complete, and watch the reserved node with 'top' or a similar program. If for example, the job reserved only 1 core and tried to run 20 processes, you'd
see 20 processes in 'top', each at 5% CPU time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To make the code a bit more polite, you can import the os module and create a new variable from the SLURM_CPUS_ON_NODE environment variable to guide Python into starting the correct number of processes:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> cpus_reserved = int(os.environ['SLURM_CPUS_ON_NODE'])<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Rodrigo Santibáñez <rsantibanez.uchile@gmail.com><br>
<b>Date: </b>Friday, May 14, 2021 at 5:17 PM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] Exposing only requested CPUs to a job on a given node.<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<div>
<p class="MsoNormal">Hi you all,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm replying to have notifications answering this question. I have a user whose python script used almost all CPUs, but configured to use only 6 cpus per task. I reviewed the code, and it doesn't have an explicit call to multiprocessing
or similar. So the user is unaware of this behavior (and also me).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Running slurm 20.02.6<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Best!<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, May 14, 2021 at 1:37 PM Luis R. Torres <<a href="mailto:lrtorres@gmail.com">lrtorres@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Folks, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We are currently running on SLURM 20.11.6 with cgroups constraints for memory and CPU/Core. Can the scheduler only expose the requested number of CPU/Core resources to a job? We have some users that employ python scripts with the multi
processing modules, and the scripts apparently use all of the CPU/Cores in a node, despite using options to constraint a task to just a given number of CPUs. We would like several multiprocessing jobs to run simultaneously on the nodes, but not step on
each other.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The sample script I use for testing is below; I'm looking for something similar to what can be done with the GPU Gres configuration where only the number of GPUs requested are exposed to the job requesting them. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">#!/usr/bin/env python3<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">import multiprocessing<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;min-height:21px">
<span style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">def worker():<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> print("Worker on CPU #%s" % multiprocessing.current_process<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">().name)<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> result=0<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> for j in range(20):<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> result += j**2<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> print ("Result on CPU {} is {}".format(multiprocessing.curr<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">ent_process().name,result))<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> return <o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;min-height:21px">
<span style="font-family:Menlo;color:black"><o:p> </o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">if __name__ == '__main__':<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> pool = multiprocessing.Pool()<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> jobs = []<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> print ("This host exposed {} CPUs".format(multiprocessing.c<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">pu_count()))<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> for i in range(multiprocessing.cpu_count()):<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black"> p = multiprocessing.Process(target=worker, name=i).star<o:p></o:p></span></p>
<p style="margin:0in;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal">
<span style="font-family:Menlo;color:black">t()<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">----------------------------------------<br>
Luis R. Torres<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</body>
</html>