<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div dir="auto">The previous version was 18 and now I am trying to upgrade to 20, so I am well within 2 major versions. </div>
<div dir="auto" style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);">
<br>
</div>
<div dir="auto" style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);">
Regards,</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Paul Edmon <pedmon@cfa.harvard.edu><br>
<b>Sent:</b> Thursday, September 8, 2022 4:44:36 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<div>
<div style="padding-bottom:10px; padding-top:5px">
<div style="padding:12px; border:1px solid #8D3970; background-color:#F7F9FA; color:#8D3970; font-size:14px; line-height:22px; font-family:Calibri,Arial,Helvetica,sans-serif">
<strong>CAUTION:</strong> This e-mail originated outside the University of Southampton.
</div>
</div>
<div>
<p>Typically slurm only supports upgrading between 2 major versions ahead. If you are on 18.08 you likely can only go to 20.02. Then after you upgrade to 20.02 you can go to 20.11 or 21.08.</p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 9/8/22 11:38 AM, Wadud Miah wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
hi Mick,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have checked that all the compute nodes and controllers all have the same version of SLURM (20.11.9). I am indeed trying to upgrade SlurmDB first, and am getting the errors in the slurmdbd.log:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"></span>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Wadud.<br>
</div>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users
<a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of Timony, Mick <a class="x_moz-txt-link-rfc2396E" href="mailto:Michael_Timony@hms.harvard.edu">
<Michael_Timony@hms.harvard.edu></a><br>
<b>Sent:</b> 08 September 2022 16:24<br>
<b>To:</b> Slurm User Community List <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="padding-bottom:10px; padding-top:5px">
<div style="padding:12px; border:1px solid #8D3970; background-color:#F7F9FA; color:#8D3970; font-size:14px; line-height:22px; font-family:Calibri,Arial,Helvetica,sans-serif">
<strong>CAUTION:</strong> This e-mail originated outside the University of Southampton.
</div>
</div>
<div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
This thread on the forums may help:<br>
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fg%2Fslurm-users%2Fc%2FYB55Ru9rvD4&data=05%7C01%7Cw.miah%40soton.ac.uk%7Cfd25248a7e6a4fa729d308da91b20c1a%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982491141437024%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=THo3JUObIzF6EWcIlQ1OsJwUwxEAGUFeMdLuvlEKhzA%3D&reserved=0" originalsrc="https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4" shash="xZNB1nabaKDULDTR32yYUkqt2PCwfThbRQaUXoXlrHJjtBGV/KbpsBKO5a419tJNuEpqQtCODJ4ZyKWhQS7Ds6N0wUtqW6dFZpRh/6Ss1yxh+QHkJfznuZX4bTSmamNxFqHKyYOHORr+hs8i69G3rfOk1yKnaK9j0E7wh57fV6M=" originalsrc="https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4" shash="BG7e7GduVjJiwznvi3vYeuudjEX9vRJO1IhmRh0TbBpmJomIrfD9iEuV1BbCF4ZBRL2Z2qTEM4Crr549rz1nTr5d6N+T0NhVkJKBZqszvalgxUZimGjOkIuG3O3igvXLgYUjLwqVvt+bVAn28S8JcAsrYqDzh9ZxnLEjMC+rLfo=" id="LPNoLPOWALinkPreview">https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4</a><br>
</div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div class="x_x__Entity x_x__EType_OWALinkPreview x_x__EId_OWALinkPreview x_x__EReadonly_1">
</div>
<br>
<br>
It looks like you have something on your network with an older version of slurm installed. I'd check the Slurm version installed on your compute nodes and controllers.<br>
<br>
The recommended approach to upgrading is to upgrade the SlurmDB first, then the controllers, then the compute nodes. More info here:<br>
<br>
<a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Fquickstart_admin.html%23upgrade&data=05%7C01%7Cw.miah%40soton.ac.uk%7Cfd25248a7e6a4fa729d308da91b20c1a%7C4a5378f929f44d3ebe89669d03ada9d8%7C0%7C0%7C637982491141437024%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Evu1PdyvAeinb0W11Ia6NxUgOvfaITmJiVau8nak%2Fac%3D&reserved=0" originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade" shash="YHT4Ux0967ZwbNfItH/sVa0AHfQtzs9SZvHNoyKJoRCdo2IPwMreZx08OlcZWYpTMd/Kbij2BvCpF4jdtxp3IuU8hNyLqI67QKuodtDegU5snAh3BsI/TB3l9nbFZTi+RWp8+wXfNuY3XlQO6i65Jjw6shI8c6YMFuLEiWR+764=" originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade" shash="RPUFI3ooTB7UYfVReHbAGSgmkupBBb/ck0p+cT8WNHGL9UqdrI45TPqxwFIE/TGbvc27oed6k3I1JS92tc5gnB8OySIoZI+LNZfNNRy/MCt9gRfEQB5+RzedPp23cE9DoDj4fR5ShEFHoaYXbR9ODj1RT/uoDH9fmCpvGcvNbg8=" id="LPNoLPOWALinkPreview_1">https://slurm.schedmd.com/quickstart_admin.html#upgrade</a><br>
<div class="x_x__Entity x_x__EType_OWALinkPreview x_x__EId_OWALinkPreview_1 x_x__EReadonly_1">
</div>
<br>
</div>
<div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0)">
Regards</div>
<div id="x_x_Signature">
<div>
<div id="x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif; color:rgb(0,0,0)">
<div class="x_x_BodyFragment"><font size="2">
<div class="x_x_PlainText"><span style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">-- </span><br style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div class="x_x_gmail_signature" style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div dir="ltr"><span style="font-family:Helvetica">Mick Timony</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><span style="font-family:Helvetica; font-size:small; background-color:rgb(255,255,255); display:inline!important">Senior DevOps Engineer</span><br>
</span><span style="font-family:Helvetica">Harvard Medical School</span></div>
<div dir="ltr"><span style="font-family:Helvetica">--</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><br>
</span></div>
</div>
</div>
</font></div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users
<a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com">
<slurm-users-bounces@lists.schedmd.com></a> on behalf of Wadud Miah <a class="x_moz-txt-link-rfc2396E" href="mailto:W.Miah@soton.ac.uk">
<W.Miah@soton.ac.uk></a><br>
<b>Sent:</b> Thursday, September 8, 2022 10:47 AM<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">
slurm-users@lists.schedmd.com</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b> [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am attempting to upgrade from SLURM 18 to 20.11.9 and when I attempt to start slurmdbd (version 20.11.9), I get the following error messages in /var/log/slurm/slurmdbd.log:<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message<br>
</div>
<div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Any help will be greatly appreciated.</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_x_Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
----------<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Wadud Miah</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Research Computing Support<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
University of Southampton<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>