<html><head></head><body><div class="ydpb1353eebyahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div><div><br></div><div><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>Dear All, </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>Good morning </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>I am able to setup a 4 node SLURM system, I am using Ubuntu 22.04 and my SLUM is working,</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>Each of the nodes we have GPU cards, and I am abble to see the information of GPU using “Nvidia-smi”</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> but when I check for “scontrol show node-1”, not able to see any entry for “Grey” , “Gres” valuses shows as null, also in the “CfgTRES” entry also not showing the gpu based entry , I am pasting my reulsts of “scontrol show node-1” , “slurmd -C” amd “nvidia-smi” here for reference </span> </p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> "scontrol show node node-1"</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);">NodeName=node-1 Arch=x86_64 CoresPerSocket=1 </p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>CPUAlloc=0 CPUTot=72 CPULoad=0.03</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>AvailableFeatures=(null)</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>ActiveFeatures=(null)</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>Gres=(null)</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>NodeAddr=node-1 NodeHostName=node-1 Version=21.08.5</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>OS=Linux 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov <span class="ydpd6155376Apple-converted-space"> </span>2 18:01:13 UTC 2 </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>RealMemory=773685 AllocMem=0 FreeMem=770972 Sockets=72 Boards=1</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>Partitions=debug </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>BootTime=2023-11-23T09:06:28 SlurmdStartTime=2023-11-23T09:07:39</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>LastBusyTime=2023-11-23T09:07:40</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>CfgTRES=cpu=72,mem=773685M,billing=72</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>AllocTRES=</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>CapWatts=n/a</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> <span class="ydpd6155376Apple-converted-space"> </span>CurrentWatts=0 AveWatts=0</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>root@node-1:~# slurmd -C</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>NodeName=node-1 CPUs=72 Boards=1 SocketsPerBoard=2 CoresPerSocket=18 ThreadsPerCore=2 RealMemory=773685</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>UpTime=0-23:48:41</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span><br></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>root@node-1:~# nvidia-smi </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>Fri Nov 24 08:55:50 2023 </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>+---------------------------------------------------------------------------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| NVIDIA-SMI 535.129.03<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Driver Version: 535.129.03<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>CUDA Version: 12.2<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>|</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|-----------------------------------------+----------------------+----------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| GPU <span class="ydpd6155376Apple-converted-space"> </span>Name<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Persistence-M | Bus-Id <span class="ydpd6155376Apple-converted-space"> </span>Disp.A | Volatile Uncorr. ECC |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| Fan <span class="ydpd6155376Apple-converted-space"> </span>Temp<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Perf <span class="ydpd6155376Apple-converted-space"> </span>Pwr:Usage/Cap |<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Memory-Usage | GPU-Util <span class="ydpd6155376Apple-converted-space"> </span>Compute M. |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>| <span class="ydpd6155376Apple-converted-space"> </span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>MIG M. |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|=========================================+======================+======================|</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>0 <span class="ydpd6155376Apple-converted-space"> </span>Tesla V100-PCIE-16GB<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Off | 00000000:06:00.0 Off | <span class="ydpd6155376Apple-converted-space"> </span>0 |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| N/A<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>26C <span class="ydpd6155376Apple-converted-space"> </span>P0 <span class="ydpd6155376Apple-converted-space"> </span>23W / 250W | <span class="ydpd6155376Apple-converted-space"> </span>4MiB / 16384MiB | <span class="ydpd6155376Apple-converted-space"> </span>0% <span class="ydpd6155376Apple-converted-space"> </span>Default |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>| <span class="ydpd6155376Apple-converted-space"> </span>| <span class="ydpd6155376Apple-converted-space"> </span>N/A |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>+-----------------------------------------+----------------------+----------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>1 <span class="ydpd6155376Apple-converted-space"> </span>Tesla V100-PCIE-16GB<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Off | 00000000:86:00.0 Off | <span class="ydpd6155376Apple-converted-space"> </span>0 |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| N/A<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>25C <span class="ydpd6155376Apple-converted-space"> </span>P0 <span class="ydpd6155376Apple-converted-space"> </span>24W / 250W | <span class="ydpd6155376Apple-converted-space"> </span>4MiB / 16384MiB | <span class="ydpd6155376Apple-converted-space"> </span>0% <span class="ydpd6155376Apple-converted-space"> </span>Default |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>| <span class="ydpd6155376Apple-converted-space"> </span>| <span class="ydpd6155376Apple-converted-space"> </span>N/A |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>+-----------------------------------------+----------------------+----------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0); min-height: 24px;"><span> </span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>+---------------------------------------------------------------------------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| Processes: <span class="ydpd6155376Apple-converted-space"> </span>|</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| <span class="ydpd6155376Apple-converted-space"> </span>GPU<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>GI<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>CI <span class="ydpd6155376Apple-converted-space"> </span>PID<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Type<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Process name <span class="ydpd6155376Apple-converted-space"> </span>GPU Memory |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| <span class="ydpd6155376Apple-converted-space"> </span>ID<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>ID<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>Usage <span class="ydpd6155376Apple-converted-space"> </span>|</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>|=======================================================================================|</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| <span class="ydpd6155376Apple-converted-space"> </span>0<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>N/A <span class="ydpd6155376Apple-converted-space"> </span>N/A <span class="ydpd6155376Apple-converted-space"> </span>2010 <span class="ydpd6155376Apple-converted-space"> </span>G<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>/usr/lib/xorg/Xorg <span class="ydpd6155376Apple-converted-space"> </span>4MiB |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>| <span class="ydpd6155376Apple-converted-space"> </span>1<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>N/A <span class="ydpd6155376Apple-converted-space"> </span>N/A <span class="ydpd6155376Apple-converted-space"> </span>2010 <span class="ydpd6155376Apple-converted-space"> </span>G<span class="ydpd6155376Apple-converted-space"> </span> <span class="ydpd6155376Apple-converted-space"> </span>/usr/lib/xorg/Xorg <span class="ydpd6155376Apple-converted-space"> </span>4MiB |</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span>+---------------------------------------------------------------------------------------+</span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><span></span></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);">Request guidance on what configuration parameters I have missed out, so that I am not able to see the GPU part in </p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);">"scontrol show node node-1”<br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);">Thanks</p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);">Joseph John </p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p style="margin: 0px; font-stretch: normal; font-size: 20px; line-height: normal; font-family: Menlo; color: rgb(0, 0, 0);"><br></p><p></p></div></div><br></div></body></html>