<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="ltr">
<div></div>
<div>
<div>
<div dir="ltr">Yes sorry Rob, I mean I did build and install with --with-nvml and it didn't find it. I then tried again specifying the location. Unfortunately, at that point users needed to run a few jobs and I wasn't able to further investigate. I will
get back at it when they've finished.</div>
</div>
<div id="ms-outlook-mobile-signature">
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Mike</div>
</div>
<div id="mail-editor-reference-message-container" class="ms-outlook-mobile-reference-message">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Groner, Rob <rug262@psu.edu><br>
<b>Sent:</b> Friday, November 11, 2022 5:07 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] NVML not found when Slurm was configured.
<div> </div>
</font></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
I'm not sure what you mean by "didn't work out for me". The error indicates slurm wasn't correctly configured for nvml when it was built, so the first step would be to get the slurm source and run configure --with-nvml and see what it says.</div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span class="ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">There's a CHANCE the error indicates slurm can't find the libnvidia-ml.so library on the system it is currently running on, so you might try installing that
package and see if slurm finds it. But I'm pretty sure it means slurm needs to be configured and built from source.</span></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span class="ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span class="ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt">Rob</span></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span class="ContentPasted0" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt"><br>
</span></div>
<div id="signature_bookmark"></div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size: 11pt;"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Lewis <mike.lewis@queensu.ca><br>
<b>Sent:</b> Friday, November 11, 2022 3:34 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] NVML not found when Slurm was configured.</font>
<div> </div>
</div>
<div lang="EN-CA" style="word-wrap:break-word">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
<div>You don't often get email from mike.lewis@queensu.ca. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; background-color: rgb(234, 234, 234); color: rgb(33, 33, 33);">
</td>
</tr>
</tbody>
</table>
<div>
<div class="x_WordSection1">
<div>
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Unfortunately this didn’t work out for me or I’m simply doing it wrong. When the current users hop off the system I’ll do some more troubleshooting. Any other insight or tips to steer me in the right direction are greatly appreciated.
</p>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;"> </span><span style="font-size: 12pt; color: black;"></span></p>
</div>
</div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;">Mike</span></p>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Lewis <mike.lewis@queensu.ca><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Friday, November 11, 2022 at 10:01 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] NVML not found when Slurm was configured.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
</div>
<div>
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;">Thanks Rob! No I just grabbed it through apt. I’ll try that now.</span><span style="font-size: 12pt; color: black;"></span></p>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;"> </span><span style="font-size: 12pt; color: black;"></span></p>
</div>
</div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;">Mike</span></p>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<div style="border:none; border-top:solid #B5C4DF 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b><span style="font-size: 12pt; color: black;">From: </span></b><span style="font-size: 12pt; color: black;">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "Groner, Rob" <rug262@psu.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Friday, November 11, 2022 at 9:32 AM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] NVML not found when Slurm was configured.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">Hi Mike,</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">I can't tell if you're compiling slurm or not on your own. You will have to if you want the functionality.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">On RedHat8, I had to install cuda-nvml-devel-11-7, so find what the equivalent is for that in Ubuntu. Basically, whatever package includes nvml.h and libnvidia-ml.so. Then, modify your configure statement when
building slurm to add "--with-nvml". Check the configure output, because it may still not find it (it didn't on our system because we installed the devel package to a non-standard location. If that's the case, you just change it to --with-nvml=<path to nvml
lib dir>. Then it should all work.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">I'll note once it's all setup, then your gres.conf becomes just "<nodenames> AutoDetect=nvml"</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">G'luck.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;">rob</span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;">
<span style="font-size: 12pt; color: black;"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="margin:0in; font-size:11pt; font-family:Calibri,sans-serif; text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<b><span style="color: black;">From:</span></b><span style="color: black;"> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Lewis <mike.lewis@queensu.ca><br>
<b>Sent:</b> Friday, November 11, 2022 9:12 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] NVML not found when Slurm was configured.</span> </p>
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
</div>
</div>
<div>
<table class="x_MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="padding: 5.25pt 1.5pt; background-color: rgb(166, 166, 166);"></td>
<td width="100%" style="width: 100%; padding: 5.25pt 3.75pt 5.25pt 11.25pt; background-color: rgb(234, 234, 234);">
<div>
<p class="x_MsoNormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="font-size: 9pt; font-family: "Segoe UI", sans-serif; color: rgb(33, 33, 33);">You don't often get email from mike.lewis@queensu.ca.
<a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable">
Learn why this is important</a></span></p>
</div>
</td>
<td width="75" style="width: 56.25pt; padding: 5.25pt 3.75pt; background-color: rgb(234, 234, 234);">
</td>
</tr>
</tbody>
</table>
<div>
<div>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
Hello Everyone,</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
New here and very new to slurm and hopefully someone can shed some light on this for me. I’m in the process of setting up a single node slurm environment with nvidia a100. I keep getting the error
<b><span style="color: rgb(224, 102, 102);">We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.</span></b> when trying to start slurmd. When removing GresTypes=gpu from slurm.conf slurmd starts
up fine and can queue up and run jobs. Cuda toolkit is installed along with NVIDIA Management Library (NVML). I went as far as removing slurm and reinstalling to see if it would pick it up. No go.
</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
OS Ubuntu 20.04, slurm.conf GresTypes=gpu is added, gres.conf AutoDetect=nvml Name=gpu Type=a100 File=/dev/nvidia0 COREs=0,1</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
I’ve searched around and see that many others have run into this but I haven’t found a fix yet. Any help would be greatly appreciated.</p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
<div>
<div>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;">Thanks,</span></p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;"> </span></p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;">Mike </span></p>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
<span style="color: black;"> </span></p>
</div>
</div>
<p class="x_xmsonormal" style="margin-top: 0px; margin-bottom: 0px;margin:0in; font-size:11pt; font-family:Calibri,sans-serif">
</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>