[slurm-users] srun and --cpus-per-task

[Hermann Schwärzler]

Hi Bjørn-Helge,
hi everone,

ok, I see. I also just re-read the documentation to find this in the 
description of the "CPUs" option:

"This can be useful when you want to schedule only the cores on a 
hyper-threaded node. If CPUs is omitted, its default will be set equal 
to the product of Boards, Sockets, CoresPerSocket, and ThreadsPerCore."

Do you happen to know if there is a difference between setting CPUs 
explicitely like you do it and not setting it but using "ThreadsPerCore=1"?

My guess is that there is no difference and in both cases only the 
physical cores are "handed out to jobs". But maybe I am wrong?

Regards,
Hermann

On 3/25/22 8:49 AM, Bjørn-Helge Mevik wrote:
> For what it's worth, we have a similar setup, with one crucial
> difference: we are handing out physical cores to jobs, not hyperthreads,
> and we are *not* seeing this behaviour:
> 
> $ srun --cpus-per-task=1 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo
> srun: job 5371678 queued and waiting for resources
> srun: job 5371678 has been allocated resources
> foo
> $ srun --cpus-per-task=3 -t 10 --mem-per-cpu=1g -A nn9999k -q devel echo foo
> srun: job 5371680 queued and waiting for resources
> srun: job 5371680 has been allocated resources
> foo
> 
> We have
> 
> SelectType=select/cons_tres
> SelectTypeParameters=CR_CPU_Memory
> 
> and node definitions like
> 
> NodeName=DEFAULT CPUs=40 Sockets=2 CoresPerSocket=20 ThreadsPerCore=2 RealMemory=182784 Gres=localscratch:330G Weight=1000
> 
> (so we set CPUs to the number of *physical cores*, not *hyperthreads*).
>