<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=big5">
</head>
<body>
<div>
<div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
The GPU nodes shouldn¡¦t have any config files ¡X they come in from the controller with configless (i.e. all config files are centralized.)</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Now, did you build Slurm on the gpu nodes, or install via package mgr? If pkg mgr, do you know if it was compiled/packaged on a node with the NVIDIA libs? (If it couldn¡¦t find the NV libs when compiled, nvml support wouldn¡¦t be built...)</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
</div>
<div><br>
</div>
<div class="ms-outlook-ios-signature"></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of David Henkemeyer <david.henkemeyer@gmail.com><br>
<b>Sent:</b> Friday, May 7, 2021 8:31:16 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Configless mode enabling issue</font>
<div> </div>
</div>
<div>
<div dir="ltr">Thank you for the reply, Will!
<div><br>
</div>
<div>The slurm.conf file only has one line in it:</div>
<div><br>
</div>
<div>AutoDetect=nvml<br>
</div>
<div><br>
</div>
<div>During my debug, I copied this file from the GPU node to the controller. But, that's when I noticed that the node w/o a GPU then crashed on startup.</div>
<div><br>
</div>
<div>David </div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, May 7, 2021 at 12:14 PM Will Dennis <<a href="mailto:wdennis@nec-labs.com">wdennis@nec-labs.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div>
<div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">Hi David,</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)"><br>
</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">What is the gres.conf on the controller¡¦s /etc/slurm ? Is it autodetect via nvml?</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)"><br>
</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">In configless the slurm.conf, gres.conf, etc is just maintained on the controller, and the worker nodes get it from there automatically (you don¡¦t want those files on the worker nodes.)
If you need to see what the slurmd daemon is seeing/doing in real-time, start slurmd on the node via ¡§slurmd-Dvvvv¡¨ and you will see the log mssgs on stdout. (If it normally runs via systemd, then ¡§systemctl stop slurmd¡¨ 1st.)</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)"><br>
</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">Regards,</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)">Will</div>
<div dir="ltr" style="color:rgb(0,0,0); background-color:rgb(255,255,255)"><br>
</div>
</div>
<div><br>
</div>
<div></div>
</div>
<hr style="display:inline-block; width:98%">
<div id="x_gmail-m_-8609344582257066279divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
on behalf of David Henkemeyer <<a href="mailto:david.henkemeyer@gmail.com" target="_blank">david.henkemeyer@gmail.com</a>><br>
<b>Sent:</b> Friday, May 7, 2021 2:41:41 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a> <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] Configless mode enabling issue</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
Hello all. My team is enabling slurm (version 20.11.5) in our environment, and we got a controller up and running, along with 2 nodes. Everything was working fine. However, when we try to enable configless mode, I ran into a problem. The node that has a
GPU is coming up in "drained" state, and sinfo -Nl shows the following:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">(dhenkemeyer)-(devops1)-(x86_64-redhat-linux-gnu)-(~/slurm/bin)<br style="box-sizing:inherit">(! 726)-> sinfo -Nl<br style="box-sizing:inherit">Fri May 07 10:20:20 2021<br style="box-sizing:inherit">NODELIST NODES PARTITION STATE CPUS S:c:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON<br style="box-sizing:inherit">devops2 1 debug* idle 4 1:4:1 9913 0 1 avx,cent none<br style="box-sizing:inherit">devops3 1 debug* drained 8 2:4:1 40213 0 1 foo,bar gres/gpu count repor</pre>
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
As you can see, it appears to be related to the gres/gpu count. Here is the entry for the node, in the slurm.conf file (which is attached) on the controller:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:kepler:1</pre>
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
Prior to this, we also tried a simpler way of expressing Gres:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:1</pre>
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
But that also failed.<span style="box-sizing:inherit; display:block; height:8px"></span>I am logging on the controller, and have enabled debug output when I launch slurmd on the nodes. On the problematic node (the one with a GPU), I am seeing this repeating
message:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: debug: Unable to register with slurm controller, retrying</pre>
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
and on the controller, I am seeing this repeating message:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">[2021-05-07T10:23:30.417] error: _slurm_rpc_node_registration node=devops3: Invalid argument</pre>
<div style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
So they are definitely related. Any help would be appreciated. I tried moving the slurm.conf file from the GPU node to the controller, but that caused our non-GPU node to puke on startup:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm¢xslurmd: debug: Unable to register with sl<br style="box-sizing:inherit"> was configured. </pre>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>