<div dir="ltr">Thank you for the reply, Will!<div><br></div><div>The slurm.conf file only has one line in it:</div><div><br></div><div>AutoDetect=nvml<br></div><div><br></div><div>During my debug, I copied this file from the GPU node to the controller. But, that's when I noticed that the node w/o a GPU then crashed on startup.</div><div><br></div><div>David </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 7, 2021 at 12:14 PM Will Dennis <<a href="mailto:wdennis@nec-labs.com">wdennis@nec-labs.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
Hi David,</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
What is the gres.conf on the controller’s /etc/slurm ? Is it autodetect via nvml?</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
In configless the slurm.conf, gres.conf, etc is just maintained on the controller, and the worker nodes get it from there automatically (you don’t want those files on the worker nodes.) If you need to see what the slurmd daemon is seeing/doing in real-time,
start slurmd on the node via “slurmd-Dvvvv” and you will see the log mssgs on stdout. (If it normally runs via systemd, then “systemctl stop slurmd” 1st.)</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
Regards,</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
Will</div>
<div dir="ltr" style="color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
</div>
<div><br>
</div>
<div></div>
</div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-8609344582257066279divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of David Henkemeyer <<a href="mailto:david.henkemeyer@gmail.com" target="_blank">david.henkemeyer@gmail.com</a>><br>
<b>Sent:</b> Friday, May 7, 2021 2:41:41 PM<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a> <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] Configless mode enabling issue</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
Hello all. My team is enabling slurm (version 20.11.5) in our environment, and we got a controller up and running, along with 2 nodes. Everything was working fine. However, when we try to enable configless mode, I ran into a problem. The node that has a
GPU is coming up in "drained" state, and sinfo -Nl shows the following:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">(dhenkemeyer)-(devops1)-(x86_64-redhat-linux-gnu)-(~/slurm/bin)<br style="box-sizing:inherit">(! 726)-> sinfo -Nl<br style="box-sizing:inherit">Fri May 07 10:20:20 2021<br style="box-sizing:inherit">NODELIST NODES PARTITION STATE CPUS S:c:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON<br style="box-sizing:inherit">devops2 1 debug* idle 4 1:4:1 9913 0 1 avx,cent none<br style="box-sizing:inherit">devops3 1 debug* drained 8 2:4:1 40213 0 1 foo,bar gres/gpu count repor</pre>
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
As you can see, it appears to be related to the gres/gpu count. Here is the entry for the node, in the slurm.conf file (which is attached) on the controller:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:kepler:1</pre>
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
Prior to this, we also tried a simpler way of expressing Gres:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:1</pre>
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
But that also failed.<span style="box-sizing:inherit;display:block;height:8px"></span>I am logging on the controller, and have enabled debug output when I launch slurmd on the nodes. On the problematic node (the one with a GPU),
I am seeing this repeating message:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: debug: Unable to register with slurm controller, retrying</pre>
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
and on the controller, I am seeing this repeating message:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">[2021-05-07T10:23:30.417] error: _slurm_rpc_node_registration node=devops3: Invalid argument</pre>
<div style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">
So they are definitely related. Any help would be appreciated. I tried moving the slurm.conf file from the GPU node to the controller, but that caused our non-GPU node to puke on startup:<br style="box-sizing:inherit">
</div>
<pre style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm│slurmd: debug: Unable to register with sl<br style="box-sizing:inherit"> was configured. </pre>
</div>
</div>
</div>
</blockquote></div>