<div dir="ltr"><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Hello all. My team is enabling slurm (version 20.11.5) in our environment, and we got a controller up and running, along with 2 nodes. Everything was working fine. However, when we try to enable configless mode, I ran into a problem. The node that has a GPU is coming up in "drained" state, and sinfo -Nl shows the following:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">(dhenkemeyer)-(devops1)-(x86_64-redhat-linux-gnu)-(~/slurm/bin)<br style="box-sizing:inherit">(! 726)-> sinfo -Nl<br style="box-sizing:inherit">Fri May 07 10:20:20 2021<br style="box-sizing:inherit">NODELIST NODES PARTITION STATE CPUS S:c:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON<br style="box-sizing:inherit">devops2 1 debug* idle 4 1:4:1 9913 0 1 avx,cent none<br style="box-sizing:inherit">devops3 1 debug* drained 8 2:4:1 40213 0 1 foo,bar gres/gpu count repor</pre><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">As you can see, it appears to be related to the gres/gpu count. Here is the entry for the node, in the slurm.conf file
(which is attached)
on the controller:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:kepler:1</pre><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">Prior to this, we also tried a simpler way of expressing Gres:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:1</pre><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">But that also failed.<span class="gmail-c-mrkdwn__br" style="box-sizing:inherit;display:block;height:8px"></span>I am logging on the controller, and have enabled debug output when I launch slurmd on the nodes. On the problematic node (the one with a GPU), I am seeing this repeating message:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: debug: Unable to register with slurm controller, retrying</pre><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">and on the controller, I am seeing this repeating message:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">[2021-05-07T10:23:30.417] error: _slurm_rpc_node_registration node=devops3: Invalid argument</pre><div class="gmail-p-rich_text_section" style="box-sizing:inherit;color:rgb(29,28,29);font-family:Slack-Lato,appleLogo,sans-serif;font-size:15px;font-variant-ligatures:common-ligatures;background-color:rgb(248,248,248)">So they are definitely related. Any help would be appreciated. I tried moving the slurm.conf file from the GPU node to the controller, but that caused our non-GPU node to puke on startup:<br style="box-sizing:inherit"></div><pre class="gmail-c-mrkdwn__pre" style="box-sizing:inherit;margin-top:4px;margin-bottom:4px;padding:8px;font-size:12px;line-height:1.50001;font-variant-ligatures:none;white-space:pre-wrap;word-break:normal;border-radius:4px;color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm│slurmd: debug: Unable to register with sl<br style="box-sizing:inherit"> was configured. </pre></div>