<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
</head>
<body>
<div>
<div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Hi David,</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
What is the gres.conf on the controller$B!G(Bs /etc/slurm ? Is it autodetect via nvml?</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
In configless the slurm.conf, gres.conf, etc is just maintained on the controller, and the worker nodes get it from there automatically (you don$B!G(Bt want those files on the worker nodes.) If you need to see what the slurmd daemon is seeing/doing in real-time,
start slurmd on the node via $B!H(Bslurmd-Dvvvv$B!I(B and you will see the log mssgs on stdout. (If it normally runs via systemd, then $B!H(Bsystemctl stop slurmd$B!I(B 1st.)</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Regards,</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Will</div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
</div>
<div><br>
</div>
<div class="ms-outlook-ios-signature"></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of David Henkemeyer <david.henkemeyer@gmail.com><br>
<b>Sent:</b> Friday, May 7, 2021 2:41:41 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] Configless mode enabling issue</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
Hello all. My team is enabling slurm (version 20.11.5) in our environment, and we got a controller up and running, along with 2 nodes. Everything was working fine. However, when we try to enable configless mode, I ran into a problem. The node that has a
GPU is coming up in "drained" state, and sinfo -Nl shows the following:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">(dhenkemeyer)-(devops1)-(x86_64-redhat-linux-gnu)-(~/slurm/bin)<br style="box-sizing:inherit">(! 726)-> sinfo -Nl<br style="box-sizing:inherit">Fri May 07 10:20:20 2021<br style="box-sizing:inherit">NODELIST NODES PARTITION STATE CPUS S:c:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON<br style="box-sizing:inherit">devops2 1 debug* idle 4 1:4:1 9913 0 1 avx,cent none<br style="box-sizing:inherit">devops3 1 debug* drained 8 2:4:1 40213 0 1 foo,bar gres/gpu count repor</pre>
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
As you can see, it appears to be related to the gres/gpu count. Here is the entry for the node, in the slurm.conf file (which is attached) on the controller:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:kepler:1</pre>
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
Prior to this, we also tried a simpler way of expressing Gres:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">NodeName=devops3 Sockets=2 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=40213 Features=foo,bar Gres=gpu:1</pre>
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
But that also failed.<span class="x_gmail-c-mrkdwn__br" style="box-sizing:inherit; display:block; height:8px"></span>I am logging on the controller, and have enabled debug output when I launch slurmd on the nodes. On the problematic node (the one with a GPU),
I am seeing this repeating message:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: debug: Unable to register with slurm controller, retrying</pre>
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
and on the controller, I am seeing this repeating message:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">[2021-05-07T10:23:30.417] error: _slurm_rpc_node_registration node=devops3: Invalid argument</pre>
<div class="x_gmail-p-rich_text_section" style="box-sizing:inherit; color:rgb(29,28,29); font-family:Slack-Lato,appleLogo,sans-serif; font-size:15px; font-variant-ligatures:common-ligatures; background-color:rgb(248,248,248)">
So they are definitely related. Any help would be appreciated. I tried moving the slurm.conf file from the GPU node to the controller, but that caused our non-GPU node to puke on startup:<br style="box-sizing:inherit">
</div>
<pre class="x_gmail-c-mrkdwn__pre" style="box-sizing:inherit; margin-top:4px; margin-bottom:4px; padding:8px; font-size:12px; line-height:1.50001; font-variant-ligatures:none; white-space:pre-wrap; word-break:normal; border-radius:4px; color:rgb(29,28,29); font-family:Monaco,Menlo,Consolas,"Courier New",monospace">slurmd: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm$B("(Bslurmd: debug: Unable to register with sl<br style="box-sizing:inherit"> was configured. </pre>
</div>
</div>
</body>
</html>