Hello Experts,

I’m a new Slurm user (so please bare with me J  …).

Recently we’ve deployed Slurm version 23.11 on a very simple cluster, which consists of a Master node (acting as a Login & Slurmdbd node as well), a Compute Node which has a NVIDIA HGX A100-SXM4-40GB GPU, detected as 4 x GPU’s: GPU [0-4], and a Storage Array presenting/sharing the NFS disk (where users’ home directories will be created as well).

The problem is that I’ve never been able to run a simple/dummy batch script in a parallel way using the 4 GPU’s. In fact, running the same command “sbatch gpu-job.sh” multiple times shows that only one single job is running, while the other jobs are in a pending state:

[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh

Submitted batch job 214

[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh

Submitted batch job 215

[slurmtest@c-a100-master test-batch-scripts]$ sbatch gpu-job.sh

Submitted batch job 216

[slurmtest@c-a100-master test-batch-scripts]$ squeue

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

               216       gpu  gpu-job   slurmtest PD       0:00      1 (None)

               215       gpu  gpu-job   slurmtest PD       0:00      1 (Priority)

               214       gpu  gpu-job   slurmtest PD       0:00      1 (Priority)

               213       gpu  gpu-job   slurmtest PD       0:00      1 (Priority)

               212       gpu  gpu-job   slurmtest PD       0:00      1 (Resources)

               211       gpu  gpu-job   slurmtest R  0:14      1 c-a100-cn01

PS: CPU jobs (i.e. using the default debug partition, without call the GPU Gres) can be run in parallel. The issue with running parallel jobs is only seen when using the GPU’s as Gres.

I’ve tried many combinations of settings in gres.conf and slurm.conf, many (if not most) of these combinations would result in error messages in slurmctld and slurmd logs.

The current gres.conf and slurm.conf contents is shown below. Even though it doesn’t give errors when restarting slurmctld and slurmd services (on master and compute nodes, resp.), but as I said, it doesn’t allow jobs to be executed in parallel. Batch script contents shared below as well, in order to give more clarity on what I’m trying to do:

[root@c-a100-master slurm]# cat gres.conf | grep -v "^#"

NodeName=c-a100-cn01 AutoDetect=nvml Name=gpu Type=A100 File=/dev/nvidia[0-3]

[root@c-a100-master slurm]# cat slurm.conf | grep -v "^#" | egrep -i "AccountingStorageTRES|GresTypes|NodeName|partition"

GresTypes=gpu

AccountingStorageTRES=gres/gpu

NodeName=c-a100-cn01 Gres=gpu:A100:4 CPUs=64 Boards=1 SocketsPerBoard=1 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=515181 State=UNKNOWN

PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP

PartitionName=gpu Nodes=ALL MaxTime=10:0:0

[slurmtest@c-a100-master test-batch-scripts]$ cat gpu-job.sh

#!/bin/bash

#SBATCH --job-name=gpu-job

#SBATCH --partition=gpu

#SBATCH --nodes=1

#SBATCH --gpus-per-node=4

#SBATCH --gres=gpu:4

#SBATCH --tasks-per-node=1

#SBATCH --output=gpu_job_output.%j   # Output file name (replaces %j with job ID)

#SBATCH --error=gpu_job_error.%j     # Error file name (replaces %j with job ID)

hostname

date

sleep 40

pwd

Any help on which changes need to be made to the config files (mainly slurm.conf and gres.cong) and/or the batch script, so that multiple jobs can be in a “Running” state at the same time (in parallel) ?

Thanks in advance for your help !

Best regards,

Hafedh Kherfani