<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">I do not have experience with using NVML and MPS, but here are my thoughts.<o:p></o:p></p>
<p class="MsoNormal">What I would focus my attention is this line:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Apr 07 16:52:33 node001 slurmd[299181]: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Apparently the Slurm build you are using has not be compiled against NVML and as such it cannot use the autodetect functionality.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-- <o:p></o:p></p>
<p class="MsoNormal"><b>Davide Vanzo, PhD<o:p></o:p></b></p>
<p class="MsoNormal"><i>Computer Scientist<o:p></o:p></i></p>
<p class="MsoNormal">BioHPC – Lyda Hill Dept. of Bioinformatics<o:p></o:p></p>
<p class="MsoNormal">UT Southwestern Medical Center<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Robert Kudyba<br>
<b>Sent:</b> Tuesday, April 7, 2020 3:56 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Header lengths are longer than data received after changing SelectType & GresTypes to use MPS<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">OK when restarting slurmd on the nodes I get these errors:<o:p></o:p></p>
<div>
<p class="MsoNormal"><br>
Apr 07 16:52:33 node001 systemd[1]: Starting Slurm node daemon...<br>
Apr 07 16:52:33 node001 slurmd[299181]: Message aggregation disabled<br>
Apr 07 16:52:33 node001 slurmd[299181]: WARNING: A line in gres.conf for GRES mps has 400 more configured than expected in slurm.conf. Ignoring extra GRES.<br>
Apr 07 16:52:33 node001 slurmd[299181]: fatal: We were configured to autodetect nvml functionality, but we weren't able to find that lib when Slurm was configured.<br>
Apr 07 16:52:33 node001 systemd[1]: slurmd.service: control process exited, code=exited status=1<br>
Apr 07 16:52:33 node001 systemd[1]: Failed to start Slurm node daemon.<br>
Apr 07 16:52:33 node001 systemd[1]: Unit slurmd.service entered failed state.<br>
Apr 07 16:52:33 node001 systemd[1]: slurmd.service failed.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Apr 07 16:43:27 node002 slurmd[273406]: error: GresPlugins changed from gpu,mic to gpu,mic,mps ignored<br>
Apr 07 16:43:27 node002 slurmd[273406]: error: Restart the slurmctld daemon to change GresPlugins<br>
Apr 07 16:43:27 node002 slurmd[273406]: error: Ignoring gres.conf record, invalid name: mps<br>
Apr 07 16:44:06 node002 slurmd[273406]: error: select_g_select_jobinfo_unpack: select plugin cons_tres not found<br>
Apr 07 16:44:06 node002 slurmd[273406]: error: select_g_select_jobinfo_unpack: unpack error<br>
Apr 07 16:44:06 node002 slurmd[273406]: error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received<br>
Apr 07 16:44:06 node002 slurmd[273406]: error: slurm_receive_msg_and_forward: Header lengths are longer than data received<br>
Apr 07 16:44:06 node002 slurmd[273406]: error: service_connection: slurm_receive_msg: Header lengths are longer than dat...ceived<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">so that " WARNING: A line in gres.conf for GRES mps has 400" must come from this entry in gres.conf:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">NodeName=node[001-003] Name=gpu Type=v100 File=/dev/nvidia0<br>
# END AUTOGENERATED SECTION -- DO NOT REMOVE<br>
Name=mps Count=400<br>
AutoDetect=nvml<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Perhaps I'm misunderstanding the Count option?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Apr 7, 2020 at 4:34 PM Davide Vanzo <<a href="mailto:Davide.Vanzo@utsouthwestern.edu">Davide.Vanzo@utsouthwestern.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Robert,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">That error is typically due to slurmd/slurmctld version mismatch or different configuration. I would not be surprised if you need to restart slurmd too after changing the SelectType
configuration.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also, do not forget this warning from the documentation when it comes to modifying SelectType:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:36.0pt">
<i>Changing this value can only be done by restarting the slurmctld daemon and will result in the loss of all job information (running and pending) since the job state save format used by each plugin is different.</i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>Davide Vanzo, PhD</b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><i>Computer Scientist</i><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">BioHPC – Lyda Hill Dept. of Bioinformatics<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">UT Southwestern Medical Center<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Robert Kudyba<br>
<b>Sent:</b> Tuesday, April 7, 2020 3:26 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] Header lengths are longer than data received after changing SelectType & GresTypes to use MPS<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#BDD6EE">
<b><span style="font-size:12.0pt;color:black">EXTERNAL MAIL</span></b><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Using Slurm 20.02 on CentIOS 7.7 with Bright Cluster. We changed the following options to enable MPS:<br>
SelectType=select/cons_tres<br>
GresTypes=gpu,mic,mps<br>
<br>
I restarted slurmctld and ran scontrol reconfigure, however all jobs get the below error:<br>
[2020-04-07T15:29:00.741] debug: backfill: no jobs to backfill<br>
[2020-04-07T15:29:03.051] Resending TERMINATE_JOB request JobId=3056 Nodelist=node[001-002]<br>
[2020-04-07T15:29:03.051] Resending TERMINATE_JOB request JobId=3061 Nodelist=node003<br>
[2020-04-07T15:29:03.051] debug: sched: Running job scheduler<br>
[2020-04-07T15:29:03.063] agent/is_node_resp: node:node003 RPC:REQUEST_TERMINATE_JOB : Header lengths are longer than data received<br>
[2020-04-07T15:29:03.071] agent/is_node_resp: node:node002 RPC:REQUEST_TERMINATE_JOB : Header lengths are longer than data received<br>
[2020-04-07T15:29:03.071] agent/is_node_resp: node:node001 RPC:REQUEST_TERMINATE_JOB : Header lengths are longer than data received<br>
<br>
Do any other options need changing? What causes these header length errors? <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:12.0pt;background:#BDD6EE">
<b><span style="font-size:12.0pt;color:black">CAUTION: </span></b><span style="font-size:12.0pt;color:black">This email originated from outside UTSW. Please be cautious of links or attachments, and validate the sender's email address before replying.</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p align="right" style="text-align:right"><b><span style="font-size:17.0pt;font-family:"Helvetica",sans-serif;color:#0054A0;letter-spacing:-2.25pt">UT</span></b><b><span style="font-size:17.0pt;font-family:"Helvetica",sans-serif;color:#0054A0"> Southwestern</span></b>
<o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p align="right" style="text-align:right"><span style="font-size:13.0pt;font-family:"Helvetica",sans-serif;color:#666D70">Medical Center</span><o:p></o:p></p>
</td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm"></td>
</tr>
<tr>
<td style="padding:0cm 0cm 0cm 0cm">
<p align="right" style="text-align:right"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#666D70">The future of medicine, today.</span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
</div>
</body>
</html>