[slurm-users] GPU configuration not working

[Paul Raines]

After a complete shutdown and restart of all daemons, things have changed
somewhat

# scontrol show nodes | egrep '(^Node|Gres)'
NodeName=mlscgpu1 Arch=x86_64 CoresPerSocket=16
    Gres=gpu:quadro_rtx_6000:10(S:0)
NodeName=mlscgpu2 Arch=x86_64 CoresPerSocket=16
    Gres=gpu:quadro_rtx_6000:5(S:0)

and I can submit like this

mlscgpu1[0]:~$ salloc -n1 -c3 -p batch --gres=gpu:quadro_rtx_6000:1 --mem=1G
salloc: Granted job allocation 16
mlscgpu1[0]:~$ printenv | grep -i CUDA
mlscgpu1[0]:~$ printenv | grep -i slurm
SLURM_NODELIST=mlscgpu1
SLURM_JOB_NAME=bash
SLURM_NODE_ALIASES=(null)
SLURM_JOB_QOS=normal
SLURM_NNODES=1
SLURM_JOBID=16
SLURM_NTASKS=1
SLURM_TASKS_PER_NODE=1
SLURM_CONF=/etc/slurm/slurm.conf
SLURM_CPUS_PER_TASK=3
SLURM_JOB_ID=16
SLURM_SUBMIT_DIR=/autofs/homes/011/raines
SLURM_NPROCS=1
SLURM_JOB_NODELIST=mlscgpu1
SLURM_CLUSTER_NAME=mlsc
SLURM_JOB_CPUS_PER_NODE=4
SLURM_SUBMIT_HOST=mlscgpu1
SLURM_JOB_PARTITION=batch
SLURM_JOB_NUM_NODES=1
SLURM_MEM_PER_NODE=1024
mlscgpu1[0]:~$

But still no CUDA_VISIBLE_DEVICES is being set



On Thu, 23 Jul 2020 10:32am, Paul Raines wrote:

>
> I have two systems in my cluster with GPUs.  Their setup in slurm.conf is
>
> GresTypes=gpu
> NodeName=mlscgpu1 Gres=gpu:quadro_rtx_6000:10 CPUs=64 Boards=1
> SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=1546557
> NodeName=mlscgpu2 Gres=gpu:quadro_rtx_6000:5 CPUs=64 Boards=1
>  SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=1546557
>
> My gres.conf is simply
>
> AutoDetect=nvml
>
> When I start slurmd on mlscgpu2 for example the log shows
>
> [2020-07-23T10:05:10.619] 5 GPU system device(s) detected
> [2020-07-23T10:05:10.619] Gres Name=gpu Type=quadro_rtx_6000 Count=1 Index=0 
> ID=7696487 File=/dev/nvidia0 Cores=0-31 CoreCnt=64 Links=-1,0,2,0,0
> [2020-07-23T10:05:10.619] Gres Name=gpu Type=quadro_rtx_6000 Count=1 Index=1 
> ID=7696487 File=/dev/nvidia1 Cores=0-31 CoreCnt=64 Links=0,-1,0,0,0
> [2020-07-23T10:05:10.619] Gres Name=gpu Type=quadro_rtx_6000 Count=1 Index=2 
> ID=7696487 File=/dev/nvidia2 Cores=0-31 CoreCnt=64 Links=2,0,-1,0,0
> [2020-07-23T10:05:10.619] Gres Name=gpu Type=quadro_rtx_6000 Count=1 Index=3 
> ID=7696487 File=/dev/nvidia3 Cores=0-31 CoreCnt=64 Links=0,0,0,-1,2
> [2020-07-23T10:05:10.619] Gres Name=gpu Type=quadro_rtx_6000 Count=1 Index=4 
> ID=7696487 File=/dev/nvidia4 Cores=0-31 CoreCnt=64 Links=0,0,0,2,-1
> [2020-07-23T10:05:10.626] slurmd version 20.02.3 started
> [2020-07-23T10:05:10.627] slurmd started on Thu, 23 Jul 2020 10:05:10 -0400
> [2020-07-23T10:05:10.627] CPUs=64 Boards=1 Sockets=2 Cores=16 Threads=2 
> Memory=1546557 TmpDisk=215198 Uptime=1723215 CPUSpecList=(null) 
> FeaturesAvail=(null) FeaturesActive=(null)
>
> # scontrol show nodes | egrep '(^Node|Gres)'
> NodeName=mlscgpu1 Arch=x86_64 CoresPerSocket=16
>   Gres=gpu:10(S:0)
> NodeName=mlscgpu2 Arch=x86_64 CoresPerSocket=16
>    Gres=gpu:5(S:0)
>
> Note who Gres above does not show "quadro_rtx_6000" and what does the (S:0)
> mean?
>
> Doing a submit like this fails:
>
> $ salloc -n1 -c3 -p batch --gres=gpu:quadro_rtx_6000:1
> salloc: error: Job submit/allocate failed: Requested node configuration is 
> not available
> salloc: Job allocation 9 has been revoked.
>
> This works but there is no CUDA device allocated in environment
>
> $ salloc -n1 -c3 -p batch --gres=gpu:1
> salloc: Granted job allocation 10
> $ printenv | grep -i cuda
> $
>
>
> I have also tried changing gres.conf to and doing scontrol reconfigure
>
> AutoDetect=nvml
> Name=gpu Type=quadro_rtx_6000 File=/dev/nvidia0 Cores=0-31
> Name=gpu Type=quadro_rtx_6000 File=/dev/nvidia1 Cores=0-31
> Name=gpu Type=quadro_rtx_6000 File=/dev/nvidia2 Cores=0-31
> Name=gpu Type=quadro_rtx_6000 File=/dev/nvidia3 Cores=0-31
> Name=gpu Type=quadro_rtx_6000 File=/dev/nvidia4 Cores=0-31
>
> But this made no difference.
>
>