<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi,<div><br></div><div>We have a ubuntu server (22.04) with currently 5 GPUs (1 x l40 and 4 x rtx_a5000).</div><div>I am trying to configure slurm such that a user can select either the l40 or a5000 gpus for a particular job.</div><div>I have configured my slurm.conf and gres.conf files similar as in this old thread:</div><div><a href="https://groups.google.com/g/slurm-users/c/fc-eoHpTNwU">https://groups.google.com/g/slurm-users/c/fc-eoHpTNwU</a></div><div>I have pasted the contents of the two files below.</div><div> </div><div>Unfortunately, my node is always on “drain” and scontrol shows this error:</div><div>Reason=gres/gpu count reported lower than configured (1 < 5)</div><div><br></div><div>Any idea what I am doing wrong?</div><div>Cheers and thanks for your help!</div><div>Gregor</div><div><br></div><div>Here are my slurm.conf and gres.conf files.</div><div><div></div><blockquote type="cite"><div>AutoDetect=off</div><div>NodeName=heimdall Name=gpu Type=l40 File=/dev/nvidia0</div><div>NodeName=heimdall Name=gpu Type=a5000 File=/dev/nvidia1</div><div>NodeName=heimdall Name=gpu Type=a5000 File=/dev/nvidia2</div><div>NodeName=heimdall Name=gpu Type=a5000 File=/dev/nvidia3</div><div>NodeName=heimdall Name=gpu Type=a5000 File=/dev/nvidia4</div></blockquote><br></div><div><div></div><blockquote type="cite"><div># slurm.conf file generated by configurator.html.</div><div># Put this file on all nodes of your cluster.</div><div># See the slurm.conf man page for more information.</div><div>#</div><div>SlurmdDebug=debug2</div><div>#</div><div>ClusterName=heimdall</div><div>SlurmctldHost=localhost</div><div>MpiDefault=none</div><div>ProctrackType=proctrack/linuxproc</div><div>ReturnToService=2</div><div>SlurmctldPidFile=/var/run/slurmctld.pid</div><div>SlurmctldPort=6817</div><div>SlurmdPidFile=/var/run/slurmd.pid</div><div>SlurmdPort=6818</div><div>SlurmdSpoolDir=/var/lib/slurm/slurmd</div><div>SlurmUser=slurm</div><div>StateSaveLocation=/var/lib/slurm/slurmctld</div><div>SwitchType=switch/none</div><div>TaskPlugin=task/none</div><div>#</div><div># TIMERS</div><div>InactiveLimit=0</div><div>KillWait=30</div><div>MinJobAge=300</div><div>SlurmctldTimeout=120</div><div>SlurmdTimeout=300</div><div>Waittime=0</div><div># SCHEDULING</div><div>SchedulerType=sched/backfill</div><div>SelectType=select/cons_tres</div><div>SelectTypeParameters=CR_Core</div><div>GresTypes=gpu</div><div>#</div><div>#AccountingStoragePort=</div><div>AccountingStorageType=accounting_storage/none</div><div>JobCompType=jobcomp/none</div><div>JobAcctGatherFrequency=30</div><div>JobAcctGatherType=jobacct_gather/none</div><div>SlurmctldDebug=info</div><div>SlurmctldLogFile=/var/log/slurm/slurmctld.log</div><div>SlurmdDebug=info</div><div>SlurmdLogFile=/var/log/slurm/slurmd.log</div><div>#</div><div># COMPUTE NODES</div><div>NodeName=heimdall CPUs=128 Gres=gpu:l40:1,gpu:a5000:4 Boards=1 SocketsPerBoard=2 CoresPerSocket=32 ThreadsPerCore=2 RealMemory=773635 State=UNKNOWN</div><div>PartitionName=heimdall Nodes=ALL Default=YES MaxTime=INFINITE State=UP DefMemPerCPU=8000 DefCpuPerGPU=16</div></blockquote></div><div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div></div></div></body></html>