<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr"><div>Hi Cristobal,</div><div><br></div><div>My hunch is it is due to the default memory/CPU settings.</div><div><br></div><div>Does it work if you do</div><div><br></div><div style="margin-left:40px">srun --gres=gpu:A100:1 --cpus-per-task=1 --mem=10G nvidia-smi<br></div><br>Sean<div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, 11 Apr 2021 at 15:26, Cristóbal Navarro <<a href="mailto:cristobal.navarro.g@gmail.com">cristobal.navarro.g@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif"><strong><table style="width:100%;float:left;background-color:lemonchiffon" cellspacing="0" cellpadding="5" border="1">
<tbody>
<tr>
<td><b>UoM notice: </b>External email. Be cautious of links, attachments, or impersonation attempts</td>
</tr>
</tbody>
</table></strong><br></div><hr></div>Hi Community,</div><div>These last two days I've been trying to understand what is the cause of the "Unable to allocate resources" error I keep getting when specifying --gres=... in a srun command (or sbatch). It fails with the error</div><div><span style="font-family:monospace">➜ srun --gres=gpu:A100:1 nvidia-smi<br>srun: error: Unable to allocate resources: Requested node configuration is not available</span></div><div><br><span style="font-family:monospace"></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">log file on the master node (not the compute one) <br></font></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif"><span style="font-family:monospace">➜ tail -f /var/log/slurm/slurmctld.log</span></font></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif"><span style="font-family:monospace">[2021-04-11T01:12:23.270] gres:gpu(7696487) type:(null)(0) job:1317 flags: state<br>[2021-04-11T01:12:23.270] gres_per_node:1 node_cnt:0<br>[2021-04-11T01:12:23.270] ntasks_per_gres:65534<br>[2021-04-11T01:12:23.270] select/cons_res: common_job_test: no job_resources info for JobId=1317 rc=-1<br>[2021-04-11T01:12:23.270] select/cons_res: common_job_test: no job_resources info for JobId=1317 rc=-1<br>[2021-04-11T01:12:23.270] select/cons_res: common_job_test: no job_resources info for JobId=1317 rc=-1<br>[2021-04-11T01:12:23.271] _pick_best_nodes: JobId=1317 never runnable in partition gpu<br>[2021-04-11T01:12:23.271] _slurm_rpc_allocate_resources: Requested node configuration is not available </span><br></font></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif"><br></font></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">If launched without --gres, it allocates all GPUs by default and nvidia-smi does work, in fact our CUDA programs do work via SLURM if --gres is not specified.<br></font></span></div><div><span style="font-family:monospace">➜ TUT04-GPU-multi git:(master) ✗ srun nvidia-smi <br>Sun Apr 11 01:05:47 2021 <br>+-----------------------------------------------------------------------------+<br>| NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.0 |<br>|-------------------------------+----------------------+----------------------+<br>| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>| | | MIG M. |<br>|===============================+======================+======================|<br>| 0 A100-SXM4-40GB On | 00000000:07:00.0 Off | 0 |<br>| N/A 31C P0 51W / 400W | 0MiB / 40537MiB | 0% Default |<br>| | | Disabled |</span></div><div><span style="font-family:monospace">....</span></div><div><span style="font-family:monospace">....</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">There is only one DGX A100 Compute node with 8 GPUs and 2x 64-core CPUs, and the gres.conf file simply is (also tried the commented lines):<br></font></span></div><div><span style="font-family:monospace">➜ ~ cat /etc/slurm/gres.conf<br># GRES configuration for native GPUS<br># DGX A100 8x Nvidia A100<br>#AutoDetect=nvml<br>Name=gpu Type=A100 File=/dev/nvidia[0-7]<br><br>#Name=gpu Type=A100 File=/dev/nvidia0 Cores=0-7<br>#Name=gpu Type=A100 File=/dev/nvidia1 Cores=8-15<br>#Name=gpu Type=A100 File=/dev/nvidia2 Cores=16-23<br>#Name=gpu Type=A100 File=/dev/nvidia3 Cores=24-31<br>#Name=gpu Type=A100 File=/dev/nvidia4 Cores=32-39<br>#Name=gpu Type=A100 File=/dev/nvidia5 Cores=40-47<br>#Name=gpu Type=A100 File=/dev/nvidia6 Cores=48-55<br>#Name=gpu Type=A100 File=/dev/nvidia7 Cores=56-63</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">Some relevant parts of the slurm.conf file</font></span></div><div><span style="font-family:monospace">➜ cat /etc/slurm/slurm.conf<br></span></div><div><span style="font-family:monospace">...<br></span></div><div><span style="font-family:monospace">## GRES<br></span></div><div><span style="font-family:monospace">GresTypes=gpu<br>AccountingStorageTRES=gres/gpu<br>DebugFlags=CPU_Bind,gres<br></span></div><div><span style="font-family:monospace">...<br></span></div><div><span style="font-family:monospace">## Nodes list<br>## Default CPU layout, native GPUs<br>NodeName=nodeGPU01 SocketsPerBoard=8 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=1024000 State=UNKNOWN Gres=gpu:A100:8 Feature=ht,gpu<br>...<br>## Partitions list<br>PartitionName=gpu OverSubscribe=FORCE MaxCPUsPerNode=128 MaxTime=INFINITE State=UP Nodes=nodeGPU01 Default=YES <br>PartitionName=cpu OverSubscribe=FORCE MaxCPUsPerNode=128 MaxTime=INFINITE State=UP Nodes=nodeGPU01 <br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">Any ideas where should I check?<br></font></span></div><div><span style="font-family:monospace"><font face="arial,sans-serif">thanks in advance</font><br></span></div><div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Cristóbal A. Navarro<br></div></div></div></div></div></div></div></div>
</blockquote></div>