<div dir="ltr"><div dir="ltr">On Wed, Apr 8, 2020 at 10:23 AM Eric Berquist <<a href="mailto:berquist@isi.edu">berquist@isi.edu</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">



<div style="overflow-wrap: break-word;">
I just ran into this issue. Specifically, SLURM looks for the NVML header file, which comes with CUDA or DCGM, in addition to the library that comes with the drivers. The check is at <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_SchedMD_slurm_blob_a763a008b7700321b51aad2e619deab00638a379_auxdir_x-5Fac-5Fnvml.m4-23L32&d=DwMGaQ&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=qEOa0Nb_0rsKW6mzpoQAwGyto3-n4jARKzCmjRt0u9E&s=El29FlTWFiGAEFiZJ4zk49gehqpwvgojL_2jGpK6Etg&e=" target="_blank">https://github.com/SchedMD/slurm/blob/a763a008b7700321b51aad2e619deab00638a379/auxdir/x_ac_nvml.m4#L32</a>.
 Once you’ve built SLURM, it’s enough to just have the GPU drivers on the nodes where SLURM will be installed.<br></div></blockquote><div><br></div><div>So how do I get around the  "fatal: We were configured to autodetect nvml functionality" error so we can use "AutoDetect=nvml"?</div><div><br></div><div><div class="gmail-gE gmail-iv gmail-gt" style="padding:20px 0px 0px;font-size:0.875rem;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><table cellpadding="0" class="gmail-cf gmail-gJ" style="border-collapse:collapse;margin-top:0px;width:auto;font-size:0.875rem;letter-spacing:0.2px;display:block"><tbody style="display:block"><tr class="gmail-acZ" style="height:auto;display:flex"><td class="gmail-gF gmail-gK" style="white-space:nowrap;padding:0px;vertical-align:top;width:531.102px;line-height:20px;display:block;max-height:20px"><table cellpadding="0" class="gmail-cf gmail-ix" style="border-collapse:collapse;table-layout:fixed;width:530.909px"><tbody><tr><td class="gmail-c2" style="display:flex"><h3 class="gmail-iw" style="overflow:hidden;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;letter-spacing:0.3px;color:rgb(95,99,104);line-height:20px"><span class="gmail-qu" tabindex="-1"><span name="Chris Samuel" class="gmail-gD" style="color:rgb(32,33,36);font-size:0.875rem;font-weight:bold;display:inline;vertical-align:top;letter-spacing:0.2px;line-height:20px">Chris Samuel</span> <span class="gmail-go" style="vertical-align:top;color:rgb(85,85,85)"><a href="mailto:chris@csamuel.org">chris@csamuel.org</a></span> <span class="gmail-go" style="vertical-align:top;color:rgb(85,85,85)"><a target="_blank" href="https://support.google.com/mail/answer/1311182?hl=en&authuser=1" class="gmail-acP" style="vertical-align:top;color:rgb(34,34,34)">via</a> <a href="http://lists.schedmd.com">lists.schedmd.com</a> </span></span></h3></td></tr></tbody></table></td></tr></tbody></table></div><div class="gmail-" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div id="gmail-:451" class="gmail-ii gmail-gt" style="font-size:0.875rem;direction:ltr;margin:8px 0px 0px;padding:0px"><div id="gmail-:4xy" class="gmail-a3s gmail-aXjCH" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif"><span class="gmail-im" style="color:rgb(80,0,80)">> Once you’ve built SLURM, it’s enough to just have the GPU drivers on the<br>> nodes where SLURM will be installed.<br><br></span>>>>Yeah I checked that at the Slurm User Group - slurmd will try and dlopen() the required libraries and should gracefully deal with them not being present.>>></div><div id="gmail-:4xy" class="gmail-a3s gmail-aXjCH" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif"><br></div><div id="gmail-:4xy" class="gmail-a3s gmail-aXjCH" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif">How do I get it to "gracefully deal with them not being present"?</div></div></div></div></div></div>