<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-CA" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello Everyone,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">New here and very new to slurm and hopefully someone can shed some light on this for me. I’m in the process of setting up a single node slurm environment with nvidia a100. I keep getting the error
<b><span style="color:#E06666">We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.</span></b> when trying to start slurmd. When removing GresTypes=gpu from slurm.conf slurmd starts up fine and
can queue up and run jobs. Cuda toolkit is installed along with NVIDIA Management Library (NVML). I went as far as removing slurm and reinstalling to see if it would pick it up. No go.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">OS Ubuntu 20.04, slurm.conf GresTypes=gpu is added, gres.conf AutoDetect=nvml Name=gpu Type=a100 File=/dev/nvidia0 COREs=0,1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve searched around and see that many others have run into this but I haven’t found a fix yet. Any help would be greatly appreciated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Mike </span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"> </span><span style="font-size:12.0pt;color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>