<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Hi Mike,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
I can't tell if you're compiling slurm or not on your own. You will have to if you want the functionality.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
On RedHat8, I had to install cuda-nvml-devel-11-7, so find what the equivalent is for that in Ubuntu. Basically, whatever package includes nvml.h and libnvidia-ml.so. Then, modify your configure statement when building slurm to add "--with-nvml". Check the
configure output, because it may still not find it (it didn't on our system because we installed the devel package to a non-standard location. If that's the case, you just change it to --with-nvml=<path to nvml lib dir>. Then it should all work.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
I'll note once it's all setup, then your gres.conf becomes just "<nodenames> AutoDetect=nvml"</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
G'luck.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof ContentPasted0">
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Michael Lewis <mike.lewis@queensu.ca><br>
<b>Sent:</b> Friday, November 11, 2022 9:12 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] NVML not found when Slurm was configured.</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle17
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-family:"Calibri",sans-serif}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-CA" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding:7px 2px 7px 2px; background-color:#A6A6A6">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width:100%; background-color:#EAEAEA; padding:7px 5px 7px 15px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
<div>You don't often get email from mike.lewis@queensu.ca. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width:75px; background-color:#EAEAEA; padding:7px 5px 7px 5px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
</td>
</tr>
</tbody>
</table>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal">Hello Everyone,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">New here and very new to slurm and hopefully someone can shed some light on this for me. I’m in the process of setting up a single node slurm environment with nvidia a100. I keep getting the error
<b><span style="color:#E06666">We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.</span></b> when trying to start slurmd. When removing GresTypes=gpu from slurm.conf slurmd starts up fine and
can queue up and run jobs. Cuda toolkit is installed along with NVIDIA Management Library (NVML). I went as far as removing slurm and reinstalling to see if it would pick it up. No go.
</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">OS Ubuntu 20.04, slurm.conf GresTypes=gpu is added, gres.conf AutoDetect=nvml Name=gpu Type=a100 File=/dev/nvidia0 COREs=0,1</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">I’ve searched around and see that many others have run into this but I haven’t found a fix yet. Any help would be greatly appreciated.</p>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_MsoNormal"><span style="color:black">Thanks,</span></p>
<p class="x_MsoNormal"><span style="color:black"> </span></p>
<p class="x_MsoNormal"><span style="color:black">Mike </span><span style="font-size:12.0pt; color:black"></span></p>
<p class="x_MsoNormal"><span style="color:black"> </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<p class="x_MsoNormal"> </p>
</div>
</div>
</div>
</body>
</html>