[slurm-users] Two gpu types on one node: gres/gpu count reported lower than configured (1 < 5)

[Feng Zhang]

Try

scontrol update NodeName=heimdall state=DOWN Reason="gpu issue"

and then

scontrol update NodeName=heimdall state=RESUME

to see if it will work. Probably just SLURM daemon having a hiccup
after you made changes.

Best,

Feng

On Mon, Oct 16, 2023 at 10:43 AM Gregor Hagelueken
<hagelueken at uni-bonn.de> wrote:
>
> Hi,
>
> We have a ubuntu server (22.04) with currently 5 GPUs (1 x l40 and 4 x rtx_a5000).
> I am trying to configure slurm such that a user can select either the l40 or a5000 gpus for a particular job.
> I have configured my slurm.conf and gres.conf files similar as in this old thread:
> https://groups.google.com/g/slurm-users/c/fc-eoHpTNwU
> I have pasted the contents of the two files below.
>
> Unfortunately, my node is always on “drain” and scontrol shows this error:
> Reason=gres/gpu count reported lower than configured (1 < 5)
>
> Any idea what I am doing wrong?
> Cheers and thanks for your help!
> Gregor
>
> Here are my slurm.conf and gres.conf files.
>
> AutoDetect=off
> NodeName=heimdall Name=gpu Type=l40  File=/dev/nvidia0
> NodeName=heimdall Name=gpu Type=a5000  File=/dev/nvidia1
> NodeName=heimdall Name=gpu Type=a5000  File=/dev/nvidia2
> NodeName=heimdall Name=gpu Type=a5000  File=/dev/nvidia3
> NodeName=heimdall Name=gpu Type=a5000  File=/dev/nvidia4
>
>
> # slurm.conf file generated by configurator.html.
> # Put this file on all nodes of your cluster.
> # See the slurm.conf man page for more information.
> #
> SlurmdDebug=debug2
> #
> ClusterName=heimdall
> SlurmctldHost=localhost
> MpiDefault=none
> ProctrackType=proctrack/linuxproc
> ReturnToService=2
> SlurmctldPidFile=/var/run/slurmctld.pid
> SlurmctldPort=6817
> SlurmdPidFile=/var/run/slurmd.pid
> SlurmdPort=6818
> SlurmdSpoolDir=/var/lib/slurm/slurmd
> SlurmUser=slurm
> StateSaveLocation=/var/lib/slurm/slurmctld
> SwitchType=switch/none
> TaskPlugin=task/none
> #
> # TIMERS
> InactiveLimit=0
> KillWait=30
> MinJobAge=300
> SlurmctldTimeout=120
> SlurmdTimeout=300
> Waittime=0
> # SCHEDULING
> SchedulerType=sched/backfill
> SelectType=select/cons_tres
> SelectTypeParameters=CR_Core
> GresTypes=gpu
> #
> #AccountingStoragePort=
> AccountingStorageType=accounting_storage/none
> JobCompType=jobcomp/none
> JobAcctGatherFrequency=30
> JobAcctGatherType=jobacct_gather/none
> SlurmctldDebug=info
> SlurmctldLogFile=/var/log/slurm/slurmctld.log
> SlurmdDebug=info
> SlurmdLogFile=/var/log/slurm/slurmd.log
> #
> # COMPUTE NODES
> NodeName=heimdall CPUs=128 Gres=gpu:l40:1,gpu:a5000:4 Boards=1 SocketsPerBoard=2 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=773635 State=UNKNOWN
> PartitionName=heimdall Nodes=ALL Default=YES MaxTime=INFINITE State=UP DefMemPerCPU=8000 DefCpuPerGPU=16
>
>