<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div>
<div>
<div dir="ltr">Yep they’re installed and can get all the gpu info from smi. </div>
</div>
<div id="ms-outlook-mobile-signature">
<div><br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Thanks,</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Mike</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Dj Merrill <deej@deej.net><br>
<b>Sent:</b> Friday, November 11, 2022 3:41:56 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>; Michael Lewis <mike.lewis@queensu.ca><br>
<b>Subject:</b> Re: [slurm-users] NVML not found when Slurm was configured.</font>
<div> </div>
</div>
<div>
<div class="x_moz-cite-prefix">At the risk of being a silly question, do you have the NVidia drivers installed on the machine?</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">Can you type "nvidia-smi" at the command line and view the GPU info?<br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">-Dj</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix"><br>
</div>
<div class="x_moz-cite-prefix">On 11/11/22 15:34, Michael Lewis wrote:<br>
</div>
<blockquote type="cite">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:"Segoe UI"}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_xmsonormal, li.x_xmsonormal, div.x_xmsonormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
div.x_WordSection1
{}
-->
</style>
<div class="x_WordSection1">
<div>
<div>
<p class="x_MsoNormal">Unfortunately this didn’t work out for me or I’m simply doing it wrong. When the current users hop off the system I’ll do some more troubleshooting. Any other insight or tips to steer me in the right direction are greatly appreciated.
</p>
<p class="x_MsoNormal"><span style="color:black"> </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<p class="x_MsoNormal"><span style="color:black">Mike</span></p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF
1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">slurm-users <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of Michael Lewis <a class="x_moz-txt-link-rfc2396E" href="mailto:mike.lewis@queensu.ca">
<mike.lewis@queensu.ca></a><br>
<b>Reply-To: </b>Slurm User Community List <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Friday, November 11, 2022 at 10:01 AM<br>
<b>To: </b>Slurm User Community List <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] NVML not found when Slurm was configured.</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="color:black">Thanks Rob! No I just grabbed it through apt. I’ll try that now.</span><span style="font-size:12.0pt; color:black"></span></p>
<p class="x_MsoNormal"><span style="color:black"> </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<p class="x_MsoNormal"><span style="color:black">Mike</span></p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF
1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><span style="font-size:12.0pt; color:black">From: </span>
</b><span style="font-size:12.0pt; color:black">slurm-users <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of "Groner, Rob" <a class="x_moz-txt-link-rfc2396E" href="mailto:rug262@psu.edu">
<rug262@psu.edu></a><br>
<b>Reply-To: </b>Slurm User Community List <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Friday, November 11, 2022 at 9:32 AM<br>
<b>To: </b><a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">"slurm-users@lists.schedmd.com"</a>
<a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] NVML not found when Slurm was configured.</span></p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">Hi Mike,</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">I can't tell if you're compiling slurm or not on your own. You will have to if you want the functionality.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">On RedHat8, I had to install cuda-nvml-devel-11-7, so find what the equivalent is for that in Ubuntu. Basically, whatever package includes nvml.h and libnvidia-ml.so.
Then, modify your configure statement when building slurm to add "--with-nvml". Check the configure output, because it may still not find it (it didn't on our system because we installed the devel package to a non-standard location. If that's the case, you
just change it to --with-nvml=<path to nvml lib dir>. Then it should all work.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">I'll note once it's all setup, then your gres.conf becomes just "<nodenames> AutoDetect=nvml"</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">G'luck.</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black">rob</span></p>
</div>
<div>
<p class="x_MsoNormal" style="background:white"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center">
<hr width="100%" size="0" align="center">
</div>
<div id="x_divRplyFwdMsg">
<p class="x_MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> slurm-users
<a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of Michael Lewis <a class="x_moz-txt-link-rfc2396E" href="mailto:mike.lewis@queensu.ca">
<mike.lewis@queensu.ca></a><br>
<b>Sent:</b> Friday, November 11, 2022 9:12 AM<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">
slurm-users@lists.schedmd.com</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> [slurm-users] NVML not found when Slurm was configured.</span> </p>
<div>
<p class="x_MsoNormal"> </p>
</div>
</div>
<div>
<table class="x_MsoNormalTable" width="100%" cellspacing="0" cellpadding="0" border="0" align="left" style="width:100.0%">
<tbody>
<tr>
<td style="background:#A6A6A6; padding:5.25pt 1.5pt
5.25pt 1.5pt">
<br>
</td>
<td width="100%" style="width:100.0%; background:#EAEAEA; padding:5.25pt
3.75pt 5.25pt 11.25pt">
<div>
<p class="x_MsoNormal" style=""><span style="">You don't often get email from <a class="x_moz-txt-link-abbreviated" href="mailto:mike.lewis@queensu.ca">
mike.lewis@queensu.ca</a>. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a></span></p>
</div>
</td>
<td width="75" style="width:56.25pt; background:#EAEAEA; padding:5.25pt
3.75pt 5.25pt 3.75pt">
<br>
</td>
</tr>
</tbody>
</table>
<div>
<div>
<p class="x_xmsonormal">Hello Everyone,</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">New here and very new to slurm and hopefully someone can shed some light on this for me. I’m in the process of setting up a single node slurm environment with nvidia a100. I keep getting the error
<b><span style="color:#E06666">We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.</span></b> when trying to start slurmd. When removing GresTypes=gpu from slurm.conf slurmd starts up fine and
can queue up and run jobs. Cuda toolkit is installed along with NVIDIA Management Library (NVML). I went as far as removing slurm and reinstalling to see if it would pick it up. No go.
</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">OS Ubuntu 20.04, slurm.conf GresTypes=gpu is added, gres.conf AutoDetect=nvml Name=gpu Type=a100 File=/dev/nvidia0 COREs=0,1</p>
<p class="x_xmsonormal"> </p>
<p class="x_xmsonormal">I’ve searched around and see that many others have run into this but I haven’t found a fix yet. Any help would be greatly appreciated.</p>
<p class="x_xmsonormal"> </p>
<div>
<div>
<p class="x_xmsonormal"><span style="color:black">Thanks,</span></p>
<p class="x_xmsonormal"><span style="color:black"> </span></p>
<p class="x_xmsonormal"><span style="color:black">Mike </span></p>
<p class="x_xmsonormal"><span style="color:black"> </span></p>
</div>
</div>
<p class="x_xmsonormal"> </p>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<pre class="x_moz-signature" cols="72">
</pre>
</div>
</body>
</html>