[slurm-users] srun and --cpus-per-task

Hermann Schwärzler hermann.schwaerzler at uibk.ac.at
Thu Mar 24 14:55:35 UTC 2022

Hi Durai,

I see the same thing as you on our test-cluster that has
configured in slurm.conf

The double-foo goes away with this:
srun --cpus-per-task=1 --hint=nomultithread echo foo

Having multithreading enabled leads to imho surprising behaviour of 
Slurm. My impression is that using it makes the concept of "a CPU" in 
Slurm somewhat fuzzy. It becomes unclear and ambiguous what you get when 
using the cpu-related options of srun, sbatch and salloc: is it a 
CPU-core or is it a CPU-thread?

I think what you found is a bug.

If you run

for c in {4..1}
  echo "## $c ###"
  srun -c $c bash -c 'echo $SLURM_CPU_BIND_LIST'

you will get:

## 4 ###
## 3 ###
## 2 ###
## 1 ###

You see: requesting 4 and 3 CPUs results in the same cpu-binding as both 
need two CPU-cores with 2 threads each. In the "3" case one of it stays 
unused but of course is not free for another job.
In the "1" case I would expect to see the same binding as in the "2" 
case. If you combine the two values in the list you *do* get the same 
value but obviously it's a list of two values and this might be the 
origin of the problem.

It is probably related to what's mentioned in the documentation for 
"[...] The default is one task per node, but note that the 
--cpus-per-task option will change this default."


On 3/24/22 1:37 PM, Durai Arasan wrote:
> Hello Slurm users,
> We are experiencing strange behavior with srun executing commands twice 
> only when setting --cpus-per-task=1
> $ srun --cpus-per-task=1 --partition=gpu-2080ti echo foo
> srun: job 1298286 queued and waiting for resources
> srun: job 1298286 has been allocated resources
> foo
> foo
> This is not seen when --cpus-per-task is another value:
> $ srun --cpus-per-task=3 --partition=gpu-2080ti echo foo
> srun: job 1298287 queued and waiting for resources
> srun: job 1298287 has been allocated resources
> foo
> Also when specifying --ntasks:
> $ srun -n1 --cpus-per-task=1 --partition=gpu-2080ti echo foo
> srun: job 1298288 queued and waiting for resources
> srun: job 1298288 has been allocated resources
> foo
> Relevant slurm.conf settings are:
> SelectType=select/cons_tres
> SelectTypeParameters=CR_Core_Memory
> # example node configuration
> NodeName=slurm-bm-58 NodeAddr=xxx.xxx.xxx.xxx Procs=72 Sockets=2 
> CoresPerSocket=18 ThreadsPerCore=2 RealMemory=354566 
> Gres=gpu:rtx2080ti:8 Feature=xx_v2.38 State=UNKNOWN
> On closer of job variables in the "--cpus-per-task=1" case, the 
> following variables have wrongly acquired a value of 2 for no reason:
> Can you see what could be wrong?
> Best,
> Durai

More information about the slurm-users mailing list