<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Typically slurm only supports upgrading between 2 major versions
ahead. If you are on 18.08 you likely can only go to 20.02. Then
after you upgrade to 20.02 you can go to 20.11 or 21.08.</p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 9/8/22 11:38 AM, Wadud Miah wrote:<br>
</div>
<blockquote type="cite"
cite="mid:LO2P265MB31195E66D6D0134181AB133EDB409@LO2P265MB3119.GBRP265.PROD.OUTLOOK.COM">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
hi Mick,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
I have checked that all the compute nodes and controllers all
have the same version of SLURM (20.11.9). I am indeed trying to
upgrade SlurmDB first, and am getting the errors in the
slurmdbd.log:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<span class="x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"></span>
<div class="x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[2022-09-08T15:45:11.115]
slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header:
protocol_version 8448 not supported</div>
</div>
<div class="x_elementToProof elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">[2022-09-08T15:33:57.001]
unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error:
slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack
SLURM_PERSIST_INIT message</div>
<div class="x_elementToProof elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br>
</div>
<div class="x_elementToProof elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Regards,</div>
<div class="x_elementToProof elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Wadud.<br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on
behalf of Timony, Mick <a class="moz-txt-link-rfc2396E" href="mailto:Michael_Timony@hms.harvard.edu"><Michael_Timony@hms.harvard.edu></a><br>
<b>Sent:</b> 08 September 2022 16:24<br>
<b>To:</b> Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to
20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div dir="ltr">
<div style="padding-bottom:10px; padding-top:5px">
<div style="padding:12px; border:1px solid #8D3970;
background-color:#F7F9FA; color:#8D3970; font-size:14px;
line-height:22px;
font-family:Calibri,Arial,Helvetica,sans-serif">
<strong>CAUTION:</strong> This e-mail originated outside the
University of Southampton.
</div>
</div>
<div>
<div class="x_elementToProof"
style="font-family:Arial,Helvetica,sans-serif;
font-size:10pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
This thread on the forums may help:<br>
<br>
</div>
<div class="x_elementToProof"
style="font-family:Arial,Helvetica,sans-serif;
font-size:10pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<a
href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fg%2Fslurm-users%2Fc%2FYB55Ru9rvD4&data=05%7C01%7Cw.miah%40soton.ac.uk%7C13f4b2b736764041dc9d08da91af4672%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982479244856364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=cQGagihxp%2BD2JTZZY%2BMKVH5I%2B386oZIXbCZT9eyfTlg%3D&reserved=0"
originalsrc="https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4"
shash="BG7e7GduVjJiwznvi3vYeuudjEX9vRJO1IhmRh0TbBpmJomIrfD9iEuV1BbCF4ZBRL2Z2qTEM4Crr549rz1nTr5d6N+T0NhVkJKBZqszvalgxUZimGjOkIuG3O3igvXLgYUjLwqVvt+bVAn28S8JcAsrYqDzh9ZxnLEjMC+rLfo="
id="LPNoLPOWALinkPreview" moz-do-not-send="true">https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4</a><br>
</div>
<div class="x_elementToProof"
style="font-family:Arial,Helvetica,sans-serif;
font-size:10pt; color:rgb(0,0,0);
background-color:rgb(255,255,255)">
<div class="x__Entity x__EType_OWALinkPreview
x__EId_OWALinkPreview x__EReadonly_1">
</div>
<br>
<br>
It looks like you have something on your network with an
older version of slurm installed. I'd check the Slurm
version installed on your compute nodes and controllers.<br>
<br>
The recommended approach to upgrading is to upgrade the
SlurmDB first, then the controllers, then the compute nodes.
More info here:<br>
<br>
<a
href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fquickstart_admin.html%23upgrade&data=05%7C01%7Cw.miah%40soton.ac.uk%7C13f4b2b736764041dc9d08da91af4672%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982479244856364%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BvJQSt4tfJY616T%2BTzfbGzw4nrTFCuZTbjyuThpssnQ%3D&reserved=0"
originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade"
shash="RPUFI3ooTB7UYfVReHbAGSgmkupBBb/ck0p+cT8WNHGL9UqdrI45TPqxwFIE/TGbvc27oed6k3I1JS92tc5gnB8OySIoZI+LNZfNNRy/MCt9gRfEQB5+RzedPp23cE9DoDj4fR5ShEFHoaYXbR9ODj1RT/uoDH9fmCpvGcvNbg8="
id="LPNoLPOWALinkPreview_1" moz-do-not-send="true">https://slurm.schedmd.com/quickstart_admin.html#upgrade</a><br>
<div class="x__Entity x__EType_OWALinkPreview
x__EId_OWALinkPreview_1 x__EReadonly_1">
</div>
<br>
</div>
<div>
<div class="x_elementToProof"
style="font-family:Arial,Helvetica,sans-serif;
font-size:10pt; color:rgb(0,0,0)">
Regards</div>
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr"
style="font-size:12pt;
font-family:Calibri,Arial,Helvetica,sans-serif;
color:rgb(0,0,0)">
<div class="x_BodyFragment"><font size="2">
<div class="x_PlainText"><span
style="color:rgb(34,34,34);
font-family:arial,sans-serif; font-size:small">-- </span><br
style="color:rgb(34,34,34);
font-family:arial,sans-serif; font-size:small">
<div class="x_gmail_signature"
style="color:rgb(34,34,34);
font-family:arial,sans-serif; font-size:small">
<div dir="ltr"><span
style="font-family:Helvetica">Mick Timony</span></div>
<div dir="ltr"><span
style="font-family:Helvetica"><span
style="font-family:Helvetica;
font-size:small;
background-color:rgb(255,255,255);
display:inline!important">Senior DevOps
Engineer</span><br>
</span><span style="font-family:Helvetica">Harvard
Medical School</span></div>
<div dir="ltr"><span
style="font-family:Helvetica">--</span></div>
<div dir="ltr"><span
style="font-family:Helvetica"><br>
</span></div>
</div>
</div>
</font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of
Wadud Miah <a class="moz-txt-link-rfc2396E" href="mailto:W.Miah@soton.ac.uk"><W.Miah@soton.ac.uk></a><br>
<b>Sent:</b> Thursday, September 8, 2022 10:47 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> [slurm-users] Upgrading SLURM from 18 to
20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">p
{margin-top:0;
margin-bottom:0}</style>
<div dir="ltr">
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
I am attempting to upgrade from SLURM 18 to 20.11.9 and
when I attempt to start slurmdbd (version 20.11.9), I get
the following error messages in
/var/log/slurm/slurmdbd.log:<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header:
protocol_version 8448 not supported</div>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no
init</div>
<div>[2022-09-08T15:33:57.001] error:
slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack
SLURM_PERSIST_INIT message<br>
</div>
<div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Any help will be greatly appreciated.</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div class="x_x_elementToProof"
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_Signature">
<div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
----------<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Wadud Miah</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
Research Computing Support<br>
</div>
<div
style="font-family:Calibri,Arial,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
University of Southampton<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>