<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
hi Mick,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I have checked that all the compute nodes and controllers all have the same version of SLURM (20.11.9). I am indeed trying to upgrade SlurmDB first, and am getting the errors in the slurmdbd.log:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></span>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_elementToProof elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message</div>
<div class="x_elementToProof elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,</div>
<div class="x_elementToProof elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Wadud.<br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Timony, Mick <Michael_Timony@hms.harvard.edu><br>
<b>Sent:</b> 08 September 2022 16:24<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="padding-bottom:10px; padding-top:5px">
<div style="padding:12px; border:1px solid #8D3970; background-color:#F7F9FA; color:#8D3970; font-size:14px; line-height:22px; font-family:Calibri,Arial,Helvetica,sans-serif">
<strong>CAUTION:</strong> This e-mail originated outside the University of Southampton.
</div>
</div>
<div>
<div class="x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
This thread on the forums may help:<br>
<br>
</div>
<div class="x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fg%2Fslurm-users%2Fc%2FYB55Ru9rvD4&data=05%7C01%7Cw.miah%40soton.ac.uk%7C13f4b2b736764041dc9d08da91af4672%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982479244856364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cQGagihxp%2BD2JTZZY%2BMKVH5I%2B386oZIXbCZT9eyfTlg%3D&reserved=0" originalsrc="https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4" shash="BG7e7GduVjJiwznvi3vYeuudjEX9vRJO1IhmRh0TbBpmJomIrfD9iEuV1BbCF4ZBRL2Z2qTEM4Crr549rz1nTr5d6N+T0NhVkJKBZqszvalgxUZimGjOkIuG3O3igvXLgYUjLwqVvt+bVAn28S8JcAsrYqDzh9ZxnLEjMC+rLfo=" id="LPNoLPOWALinkPreview">https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4</a><br>
</div>
<div class="x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div class="x__Entity x__EType_OWALinkPreview x__EId_OWALinkPreview x__EReadonly_1">
</div>
<br>
<br>
It looks like you have something on your network with an older version of slurm installed. I'd check the Slurm version installed on your compute nodes and controllers.<br>
<br>
The recommended approach to upgrading is to upgrade the SlurmDB first, then the controllers, then the compute nodes. More info here:<br>
<br>
<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fquickstart_admin.html%23upgrade&data=05%7C01%7Cw.miah%40soton.ac.uk%7C13f4b2b736764041dc9d08da91af4672%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982479244856364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BvJQSt4tfJY616T%2BTzfbGzw4nrTFCuZTbjyuThpssnQ%3D&reserved=0" originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade" shash="RPUFI3ooTB7UYfVReHbAGSgmkupBBb/ck0p+cT8WNHGL9UqdrI45TPqxwFIE/TGbvc27oed6k3I1JS92tc5gnB8OySIoZI+LNZfNNRy/MCt9gRfEQB5+RzedPp23cE9DoDj4fR5ShEFHoaYXbR9ODj1RT/uoDH9fmCpvGcvNbg8=" id="LPNoLPOWALinkPreview_1">https://slurm.schedmd.com/quickstart_admin.html#upgrade</a><br>
<div class="x__Entity x__EType_OWALinkPreview x__EId_OWALinkPreview_1 x__EReadonly_1">
</div>
<br>
</div>
<div>
<div class="x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0)">
Regards</div>
<div id="x_Signature">
<div>
<div></div>
<div></div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif; color:rgb(0,0,0)">
<div class="x_BodyFragment"><font size="2">
<div class="x_PlainText"><span style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">-- </span><br style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div class="x_gmail_signature" style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div dir="ltr"><span style="font-family:Helvetica">Mick Timony</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><span style="font-family:Helvetica; font-size:small; background-color:rgb(255,255,255); display:inline!important">Senior DevOps Engineer</span><br>
</span><span style="font-family:Helvetica">Harvard Medical School</span></div>
<div dir="ltr"><span style="font-family:Helvetica">--</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><br>
</span></div>
</div>
</div>
</font></div>
</div>
</div>
</div>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Wadud Miah <W.Miah@soton.ac.uk><br>
<b>Sent:</b> Thursday, September 8, 2022 10:47 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am attempting to upgrade from SLURM 18 to 20.11.9 and when I attempt to start slurmdbd (version 20.11.9), I get the following error messages in /var/log/slurm/slurmdbd.log:<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message<br>
</div>
<div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Any help will be greatly appreciated.</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_Signature">
<div>
<div></div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
----------<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Wadud Miah</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Research Computing Support<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
University of Southampton<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>