<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif}
a:link, span.MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.msonormal0, li.msonormal0, div.msonormal0
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.p1, li.p1, div.p1
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif}
p.p2, li.p2, div.p2
{margin-right:0in;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif}
span.EmailStyle21
{font-family:"Calibri",sans-serif;
color:#1F497D}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
-->
</style><style type="text/css" id="owaParaStyle"></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Here is some more data:
<div><br>
</div>
<div>Changed slurm.conf to have </div>
<div><br>
</div>
<div>
<p class="p1"><span class="s1">SelectType=select/cons_res</span></p>
<p class="p1"><span class="s1">SelectTypeParameters=CR_CPU</span></p>
<div>
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">Then restarted</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px"> <span style="font-family: 'Times New Roman', serif; font-size: 12pt;">sudo systemctl restart slurmctld.service</span></div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">The log on the host said:</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<p class="p1"><span class="s1">[2017-11-29T12:23:56.384] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:23:56.384] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:23:56.384] error: Malformed RPC of type REQUEST_ABORT_JOB(6013) received</span></p>
<div style="font-family:Tahoma; font-size:13px"><span style="font-family: 'Times New Roman', serif; font-size: 12pt;">[2017-11-29T12:23:56.384] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">Then did a sudo scontrol reconfigure and the log said:</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<p class="p1"><span class="s1">[2017-11-29T12:23:56.394] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:24:34.889] Message aggregation disabled</span></p>
<div style="font-family:Tahoma; font-size:13px"><span style="font-family: 'Times New Roman', serif; font-size: 12pt;">[2017-11-29T12:24:34.890] Resource spec: Reserved system memory limit not configured for this node</span></div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">Sview had running jobs cleard out of its context (they are still running) But I kinda expect that.</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">I then submitted 6 jobs to the partition that do nothing but sleep and the log says:</div>
<div style="font-family:Tahoma; font-size:13px"><br>
</div>
<div style="font-family:Tahoma; font-size:13px">
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.424] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: Malformed RPC of type REQUEST_BATCH_JOB_LAUNCH(4005) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.425] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.434] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.434] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.434] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.434] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.435] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.435] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.436] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.437] error: we don't have select plugin type 101</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.437] error: select_g_select_jobinfo_unpack: unpack error</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.437] error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.437] error: slurm_receive_msg_and_forward: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.446] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.446] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.446] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.446] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.447] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:25:39.447] error: service_connection: slurm_receive_msg: Header lengths are longer than data received</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
<p class="p1"><span class="s1">Lastly changes the config back to linear and restarted reconfigured and the node log says:</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:26:19.617] [6684.0] job_manager exiting with aborted job</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:26:19.621] [6684.0] done with job</span></p>
<p class="p1"><span class="s1">[2017-11-29T12:26:24.591] Message aggregation disabled</span></p>
<p class="p1"></p>
<p class="p1"><span class="s1">[2017-11-29T12:26:24.592] Resource spec: Reserved system memory limit not configured for this node</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
</div>
<div style="font-family:Tahoma; font-size:13px">Ethan VanMatre</div>
<div style="font-family:Tahoma; font-size:13px">Informatics Research Analyst<br>
Institute on Development and Disability<br>
Oregon Health & Science University<br>
CSLU - GH40<br>
3181 SW Sam Jackson Park Rd<br>
Portland, OR 97239<br>
(503) 346-3764<br>
vanmatre@ohsu.edu<br>
</div>
</div>
</div>
</div>
</div>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px"><br>
<div>
<div class="WordSection1">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt; font-family:"Tahoma",sans-serif; color:black"></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>