<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">As I said at the beginning, I have never played with MPS, so my answer is based only on what the Slurm documentation shows.<o:p></o:p></p>
<p class="MsoNormal">Apparently MPS does not require NVML, hence you can avoid setting AutoDetect and instead list the GPU resources in the gres.conf file old style. That should help you to get over that fatal error without having to rebuild Slurm from sources.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
<p class="MsoNormal"><b>Davide Vanzo, PhD<o:p></o:p></b></p>
<p class="MsoNormal"><i>Computer Scientist<o:p></o:p></i></p>
<p class="MsoNormal">BioHPC – Lyda Hill Dept. of Bioinformatics<o:p></o:p></p>
<p class="MsoNormal">UT Southwestern Medical Center<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Robert Kudyba<br>
<b>Sent:</b> Wednesday, April 8, 2020 4:50 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Header lengths are longer than data received after changing SelectType & GresTypes to use MPS<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#BDD6EE">
<b><span style="font-size:12.0pt;color:black">EXTERNAL MAIL</span></b><o:p></o:p></p>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">> use yum install slurm20, here they show Slurm 19 but it's the same for 20<br>
<br>
In that case you'll need to open a bug with Bright to get them to <br>
rebuild Slurm with nvml support.<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">They told me they don't officially support MPS nor Slurm and to come here to get support (or pay SchedMD). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The vicious cycle continues. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Since all I want it MPS enabled from <a href="https://slurm.schedmd.com/gres.html#MPS_config_example_2">https://slurm.schedmd.com/gres.html#MPS_config_example_2</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">"CUDA Multi-Process Service (MPS) provides a mechanism where GPUs can be shared by multiple jobs, where each job is allocated some percentage of the GPU's resources. The total count of MPS resources available on a node should be configured
in the slurm.conf file (e.g. "NodeName=tux[1-16] Gres=gpu:2,mps:200"). Several options are available for configuring MPS in the gres.conf file as listed below with examples following that:<br>
<br>
No MPS configuration: The count of gres/mps elements defined in the slurm.conf will be evenly distributed across all GPUs configured on the node. For the example, "NodeName=tux[1-16] Gres=gpu:2,mps:200" will configure a count of 100 gres/mps resources on each
of the two GPUs."<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Do I even need to edit gres.conf? Can I just leave out AutoDetect=nvml?<o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#BDD6EE">
<b><span style="font-size:12.0pt;color:black">CAUTION: </span></b><span style="font-size:12.0pt;color:black">This email originated from outside UTSW. Please be cautious of links or attachments, and validate the sender's email address before replying.</span><o:p></o:p></p>
</div>
<br>
<hr>
<table border="0" cellpadding="0">
<tbody>
<tr>
<td><!-- Copyright 2015. UT Southwestern Medical Center 12082014v1 -->
<p align="right" style="text-align:right"><span style="font-size:17pt;letter-spacing:-3px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">UT</span><span style="font-size:17pt;letter-spacing:-0px;font-family:"Helvetica","sans-serif";color:#0054A0;font-weight:bolder;">
Southwestern</span> </p>
</td>
</tr>
<tr>
<td>
<p align="right" style="text-align:right"><span style="font-size:13pt;font-family:"Helvetica","sans-serif";color:#666D70;">Medical Center</span></p>
</td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td>
<p style="text-align:right"><span style="font-size:10pt;font-family:"Helvetica","sans-serif"; color:#666D70;">The future of medicine, today.</span></p>
</td>
</tr>
</tbody>
</table>
</body>
</html>