<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Perhaps there is a node that you don't have access to that's trying to access your slurmdDB. That's the scenario that occurred in the forum posting I linked to previously:<br>
<br>
<span style="font-size: 10pt; margin: 0px; color: black; background-color: white;"><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__eur03.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgroups.google.com-252Fg-252Fslurm-2Dusers-252Fc-252FYB55Ru9rvD4-26data-3D05-257C01-257Cw.miah-2540soton.ac.uk-257C13f4b2b736764041dc9d08da91af4672-257C4a5378f929f44d3ebe89669d03ada9d8-257C0-257C0-257C637982479244856364-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-257C-257C-257C-26sdata-3DcQGagihxp-252BD2JTZZY-252BMKVH5I-252B386oZIXbCZT9eyfTlg-253D-26reserved-3D0&d=DwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VdVezmCbZuLlhdKBk1emX2rlpWZ2DrL3v-wR0vX7eA4&m=YD5IvPVS42d8-mPHmPwJWday89-bLJVcIhOMV2h0RF19Z6VRQvMRwFxEojs-cw5m&s=zanl-ryBDYiUX9HUz8yAQSXuRJ2IRdJU4QxUNReMkGU&e=" target="_blank" rel="noopener noreferrer" data-auth="VerificationFailed" shash="BG7e7GduVjJiwznvi3vYeuudjEX9vRJO1IhmRh0TbBpmJomIrfD9iEuV1BbCF4ZBRL2Z2qTEM4Crr549rz1nTr5d6N+T0NhVkJKBZqszvalgxUZimGjOkIuG3O3igvXLgYUjLwqVvt+bVAn28S8JcAsrYqDzh9ZxnLEjMC+rLfo=" title="" data-linkindex="0" style="margin:0px">https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4</a></span><br>
<br>
<br>
You can try running netstat or other tools to see what IP's are connecting, or turn on debug logging for the slurmdb:<br>
<br>
<a href="https://slurm.schedmd.com/slurmdbd.conf.html#OPT_DebugFlags" id="LPlnk143321">https://slurm.schedmd.com/slurmdbd.conf.html#OPT_DebugFlags</a><br>
<br>
If everything else is running correctly, you could ignore the error.</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
--Mick</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Wadud Miah <W.Miah@soton.ac.uk><br>
<b>Sent:</b> Thursday, September 8, 2022 11:38 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
hi Mick,</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I have checked that all the compute nodes and controllers all have the same version of SLURM (20.11.9). I am indeed trying to upgrade SlurmDB first, and am getting the errors in the slurmdbd.log:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"></span>
<div class="x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,</div>
<div class="x_x_elementToProof x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Wadud.<br>
</div>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Timony, Mick <Michael_Timony@hms.harvard.edu><br>
<b>Sent:</b> 08 September 2022 16:24<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="padding-bottom:10px; padding-top:5px">
<div style="padding:12px; border:1px solid #8D3970; background-color:#F7F9FA; color:#8D3970; font-size:14px; line-height:22px; font-family:Calibri,Arial,Helvetica,sans-serif">
<strong>CAUTION:</strong> This e-mail originated outside the University of Southampton.
</div>
</div>
<div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
This thread on the forums may help:<br>
<br>
</div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__eur03.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgroups.google.com-252Fg-252Fslurm-2Dusers-252Fc-252FYB55Ru9rvD4-26data-3D05-257C01-257Cw.miah-2540soton.ac.uk-257C13f4b2b736764041dc9d08da91af4672-257C4a5378f929f44d3ebe89669d03ada9d8-257C0-257C0-257C637982479244856364-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-257C-257C-257C-26sdata-3DcQGagihxp-252BD2JTZZY-252BMKVH5I-252B386oZIXbCZT9eyfTlg-253D-26reserved-3D0&d=DwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VdVezmCbZuLlhdKBk1emX2rlpWZ2DrL3v-wR0vX7eA4&m=YD5IvPVS42d8-mPHmPwJWday89-bLJVcIhOMV2h0RF19Z6VRQvMRwFxEojs-cw5m&s=zanl-ryBDYiUX9HUz8yAQSXuRJ2IRdJU4QxUNReMkGU&e=" originalsrc="https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4" shash="BG7e7GduVjJiwznvi3vYeuudjEX9vRJO1IhmRh0TbBpmJomIrfD9iEuV1BbCF4ZBRL2Z2qTEM4Crr549rz1nTr5d6N+T0NhVkJKBZqszvalgxUZimGjOkIuG3O3igvXLgYUjLwqVvt+bVAn28S8JcAsrYqDzh9ZxnLEjMC+rLfo=" id="LPNoLPOWALinkPreview">https://groups.google.com/g/slurm-users/c/YB55Ru9rvD4</a><br>
</div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">
<div class="x_x__Entity x_x__EType_OWALinkPreview x_x__EId_OWALinkPreview x_x__EReadonly_1">
</div>
<br>
<br>
It looks like you have something on your network with an older version of slurm installed. I'd check the Slurm version installed on your compute nodes and controllers.<br>
<br>
The recommended approach to upgrading is to upgrade the SlurmDB first, then the controllers, then the compute nodes. More info here:<br>
<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__eur03.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fslurm.schedmd.com-252Fquickstart-5Fadmin.html-2523upgrade-26data-3D05-257C01-257Cw.miah-2540soton.ac.uk-257C13f4b2b736764041dc9d08da91af4672-257C4a5378f929f44d3ebe89669d03ada9d8-257C0-257C0-257C637982479244856364-257CUnknown-257CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0-253D-257C3000-257C-257C-257C-26sdata-3DBvJQSt4tfJY616T-252BTzfbGzw4nrTFCuZTbjyuThpssnQ-253D-26reserved-3D0&d=DwMFAg&c=WO-RGvefibhHBZq3fL85hQ&r=VdVezmCbZuLlhdKBk1emX2rlpWZ2DrL3v-wR0vX7eA4&m=YD5IvPVS42d8-mPHmPwJWday89-bLJVcIhOMV2h0RF19Z6VRQvMRwFxEojs-cw5m&s=qIqQ0GI-S57qI4yYWDb_ZQLiXm0rqBPIHmZgohMTprc&e=" originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade" shash="RPUFI3ooTB7UYfVReHbAGSgmkupBBb/ck0p+cT8WNHGL9UqdrI45TPqxwFIE/TGbvc27oed6k3I1JS92tc5gnB8OySIoZI+LNZfNNRy/MCt9gRfEQB5+RzedPp23cE9DoDj4fR5ShEFHoaYXbR9ODj1RT/uoDH9fmCpvGcvNbg8=" id="LPNoLPOWALinkPreview_1">https://slurm.schedmd.com/quickstart_admin.html#upgrade</a><br>
<div class="x_x__Entity x_x__EType_OWALinkPreview x_x__EId_OWALinkPreview_1 x_x__EReadonly_1">
</div>
<br>
</div>
<div>
<div class="x_x_elementToProof" style="font-family:Arial,Helvetica,sans-serif; font-size:10pt; color:rgb(0,0,0)">
Regards</div>
<div id="x_x_Signature">
<div>
<div></div>
<div></div>
<div id="x_x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif; color:rgb(0,0,0)">
<div class="x_x_BodyFragment"><font size="2">
<div class="x_x_PlainText"><span style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">-- </span><br style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div class="x_x_gmail_signature" style="color:rgb(34,34,34); font-family:arial,sans-serif; font-size:small">
<div dir="ltr"><span style="font-family:Helvetica">Mick Timony</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><span style="font-family:Helvetica; font-size:small; background-color:rgb(255,255,255); display:inline!important">Senior DevOps Engineer</span><br>
</span><span style="font-family:Helvetica">Harvard Medical School</span></div>
<div dir="ltr"><span style="font-family:Helvetica">--</span></div>
<div dir="ltr"><span style="font-family:Helvetica"><br>
</span></div>
</div>
</div>
</font></div>
</div>
</div>
</div>
</div>
<div id="x_x_appendonsend"></div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Wadud Miah <W.Miah@soton.ac.uk><br>
<b>Sent:</b> Thursday, September 8, 2022 10:47 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> [slurm-users] Upgrading SLURM from 18 to 20.11.9</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I am attempting to upgrade from SLURM 18 to 20.11.9 and when I attempt to start slurmdbd (version 20.11.9), I get the following error messages in /var/log/slurm/slurmdbd.log:<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:45:11.115] slurmdbd version 20.11.9 started
<div>[2022-09-08T15:45:23.001] error: unpack_header: protocol_version 8448 not supported</div>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[2022-09-08T15:33:57.001] unpacking header
<div>[2022-09-08T15:33:57.001] error: destroy_forward: no init</div>
<div>[2022-09-08T15:33:57.001] error: slurm_unpack_received_msg: Message receive failure</div>
[2022-09-08T15:33:57.011] error: CONN:11 Failed to unpack SLURM_PERSIST_INIT message<br>
</div>
<div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Any help will be greatly appreciated.</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Regards,<br>
</div>
<div class="x_x_x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_x_x_Signature">
<div>
<div></div>
<div></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
----------<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Wadud Miah</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Research Computing Support<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
University of Southampton<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>