[slurm-users] CPUSpecList confusion

[Paul Raines]

Hmm.  Actually looks like confusion between CPU IDs on the system
and what SLURM thinks the IDs are

# scontrol -d show job 8
...
      Nodes=foobar CPU_IDs=14-21 Mem=25600 GRES=
...

# cat 
/sys/fs/cgroup/system.slice/slurmstepd.scope/job_8/cpuset.cpus.effective
7-10,39-42


-- Paul Raines (http://help.nmr.mgh.harvard.edu)



On Tue, 13 Dec 2022 9:40am, Paul Raines wrote:

>
> Oh but that does explain the CfgTRES=cpu=14.  With the CpuSpecList
> below and SlurmdOffSpec I do get CfgTRES=cpu=50 so that makes sense.
>
> The issue remains that thought the number of cpus in CpuSpecList
> is taken into account, the exact IDs seem to be ignored.
>
>
> -- Paul Raines (http://help.nmr.mgh.harvard.edu)
>
>
>
> On Tue, 13 Dec 2022 9:34am, Paul Raines wrote:
>
>>
>>  I have tried it both ways with the same result.  The assigned CPUs
>>  will be both in and out of the range given to CpuSpecList
>>
>>  I tried setting using commas instead of ranges so used
>>
>>  CpuSpecList=0,1,2,3,4,5,6,7,8,9,10,11,12,13
>>
>>  But still does not work
>>
>>  $ srun -p basic -N 1 --ntasks-per-node=1 --mem=25G \
>>  --time=10:00:00 --cpus-per-task=8 --pty /bin/bash
>>  $ grep -i ^cpu /proc/self/status
>>  Cpus_allowed:   00000780,00000780
>>  Cpus_allowed_list:      7-10,39-42
>> 
>>
>>  -- Paul Raines (http://help.nmr.mgh.harvard.edu)
>> 
>> 
>>
>>  On Mon, 12 Dec 2022 10:21am, Sean Maxwell wrote:
>>
>>>   Hi Paul,
>>>
>>>   Nodename=foobar \
>>>>      CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16
>>>>      ThreadsPerCore=2
>>>>      \
>>>>      RealMemory=256312 MemSpecLimit=32768 CpuSpecList=14-63 \
>>>>      TmpDisk=6000000 Gres=gpu:nvidia_rtx_a6000:1
>>>>
>>>>   The slurm.conf also has:
>>>>
>>>>   ProctrackType=proctrack/cgroup
>>>>   TaskPlugin=task/affinity,task/cgroup
>>>>   TaskPluginParam=Cores,*SlurmdOf**fSpec*,Verbose
>>>> 
>>>
>>>   Doesn't setting SlurmdOffSpec tell Slurmd that is should NOT use the
>>>   CPUs
>>>   in the spec list? (
>>>   https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdOffSpec)
>>>   In this case, I believe it uses what is left, which is the 0-13. We are
>>>   just starting to work on this ourselves, and were looking at this
>>>   setting.
>>>
>>>   Best,
>>>
>>>   -Sean
>>> 
>> 
>
The information in this e-mail is intended only for the person to whom it is addressed.  If you believe this e-mail was sent to you in error and the e-mail contains patient information, please contact the Mass General Brigham Compliance HelpLine at https://www.massgeneralbrigham.org/complianceline <https://www.massgeneralbrigham.org/complianceline> .
Please note that this e-mail is not secure (encrypted).  If you do not wish to continue communication over unencrypted e-mail, please notify the sender of this message immediately.  Continuing to send or respond to e-mail after receiving this message means you understand and accept this risk and wish to continue to communicate over unencrypted e-mail.