[slurm-users] Limiting the number of CPU

[Sukman]

Hi all,

I am currently having a problem in limiting the number of CPU used for running a job.
I tried to limit the CPU to just only 2 from the maximum 56.
But, when I run the job, using only 1 CPU, the QOS has been reached already.
When I set the CPU to 56, the job runs finely.

Does anyone have any suggestion regarding this problem?


Following is the details of the problem.


My node has 56 cores (2sockets x 28cores).


I configured already slurm.conf by enabling the qos/limit enforcement.

#slurm.conf
AccountingStorageEnforce=qos,limits


For QOS itself, I just tried applying a simple limit-CPU number to be 2.

#QOS
sacctmgr show qos where Name=normal_compute format=Name,Priority,UsageFactor,MaxWall,MaxTRESPU
      Name   Priority UsageFactor     MaxWall     MaxTRESPU 
---------- ---------- ----------- ----------- ------------- 
normal_co+         10    1.000000    00:01:00  cpu=2,mem=1G


I then applied the QOS to a specific user, sukman.

#QOS-defined user
sacctmgr list association where User=sukman format=User,QOS,
      User                  QOS 
---------- -------------------- 
    sukman       normal_compute


Then, I tried to run a simple bash command, hostname, by just using 1 node, 1 task, and 1 CPU

#!/bin/bash
#SBATCH --job-name=hostname
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --nodelist=cn110

srun hostname


However, the QOS has been reached already.

squeue 
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                68      defq hostname   sukman PD       0:00      1 (QOSMaxCpuPerUserLimit)


When I change the CPU limit to the max cores number in a server, 56 cores

sacctmgr show qos where Name=normal_compute format=Name,Priority,UsageFactor,MaxWall,MaxTRESPU
      Name   Priority UsageFactor     MaxWall     MaxTRESPU 
---------- ---------- ----------- ----------- ------------- 
normal_co+         10    1.000000    00:01:00 cpu=56,mem=1G


the script runs perfectly.

cat slurm-68.out 
cn110



--------------------------------

Suksmandhira H
ITB Indonesia