<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Sid,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
On our cluster, it performs just like your PBS cluster.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
$ srun -N 1 --cpus-per-task 8 --time 01:00:00 --mem 2g --partition physicaltest -q hpcadmin --pty python3
<div>srun: job 27060036 queued and waiting for resources</div>
<div>srun: job 27060036 has been allocated resources</div>
<div>Python 3.6.8 (default, Aug 13 2020, 07:46:32)</div>
<div>[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>
<div>>>> import os</div>
<div>>>> os.cpu_count()</div>
<div>72</div>
<div>>>> len(os.sched_getaffinity(0))</div>
8</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We do have cgroups set up to limit which CPUs a user has access to. We do it with</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
ProctrackType = proctrack/cgroup<br>
TaskPlugin = task/affinity,task/cgroup</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sean<br>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Sid Young <sid.young@gmail.com><br>
<b>Sent:</b> Friday, 18 June 2021 14:20<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [EXT] [slurm-users] incorrect number of cpu's being reported in srun job</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div class="x_mc-ip-hide">
<div style="color:#000000; font-size:12px; text-align:left; font-family:Helvetica,Arial,sans-serif">
<strong>
<table style="width:100%; float:left" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="color:red"><b>External email: </b>Please exercise caution</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
G'Day all,</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I've had a question from a user of our new HPC, the following should explain it:<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
➜ srun -N 1 --cpus-per-task 8 --time 01:00:00 --mem 2g --pty python3
<div>Python 3.6.8 (default, Nov 16 2020, 16:55:22)</div>
<div>[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>
<div>>>> import os</div>
<div>>>> os.cpu_count()</div>
<div>256</div>
<div>>>> len(os.sched_getaffinity(0))</div>
<div>256</div>
>>><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The output of os.cpu_count() is correct: there are 256 CPUs on the server, but the output of len(os.sched_getaffinity(0)) is still 256 when I was expecting it to be 8 - the number of CPUs this process is restricted to. Is my slurm command incorrect? When I
run a similar test on XXXXXX I get the expected behaviour:<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
➜ qsub -I -l select=1:ncpus=4:mem=1gb
<div>qsub: waiting for job 9616042.pbs to start</div>
<div>qsub: job 9616042.pbs ready</div>
<div>➜ python3</div>
<div>Python 3.4.10 (default, Dec 13 2019, 16:20:47) [GCC] on linux</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>
<div>>>> import os</div>
<div>>>> os.cpu_count()</div>
<div>72</div>
<div>>>> len(os.sched_getaffinity(0))</div>
<div>4</div>
>>><br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This seems to be a problem for me as I have a program provided by a third-party company that keeps trying to run with 256 threads and crashes. The program is a compiled binary so I don't know if they're just grabbing the number of CPUs or correctly getting
the scheduler affinity, but it seems as though TRI's HPC will return the total number of CPUs in any case. There aren't any options with the program to set the number of threads manually.</div>
<br class="x_gmail-Apple-interchange-newline">
</div>
<div>My question to the group is what's causing this? Do I need a cgroups plugin?</div>
<div><br>
</div>
<div>I think these are the relevant lines from the slurm.conf file:</div>
<div><br>
</div>
<div>SelectType=select/cons_res<br>
SelectTypeParameters=CR_CPU_Memory<br>
ReturnToService=1<br>
CpuFreqGovernors=OnDemand,Performance,UserSpace<br>
CpuFreqDef=Performance<br>
</div>
<div><br>
</div>
<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
<div>Sid Young</div>
<div>Translational Research Institute<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>