<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Whops - this was accidentially sent way before it was finished. Sorry for the noise :-)</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>/Esben</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> EPF (Esben Peter Friis)<br>
<b>Sent:</b> Thursday, June 2, 2022 08:16<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Segfault in slurmctld 22.05 </font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi all</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We installed Slurm 22.05 yesterday, and the slurmctld deamon crashes randomly every couply of hours. It's not much information I can get out of it, but running slurmctld in foreground (slurmctld -D -vvvvv) does not reveal anything before the crash.</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Possibly unrelated errors reported by slurmctld:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] We need to forward this to other nodes use slurm_receive_msg_and_forward instead
<div>slurmctld: error: auth_g_unpack: authentication plugin unknown(1297436231) not found</div>
<div>slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] auth_g_unpack: 0 has authentication error: No error</div>
<div>slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] Header lengths are longer than data received</div>
<div>slurmctld: error: slurm_receive_msg [127.0.0.1:43924]: Unspecified error</div>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
dmesg shows:</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
[87848.891824] sched_agent[163673]: segfault at 12 ip 00007fabd0f1986b sp 00007fabca1843b0 error 4
<div>[87848.891827] Code: 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 48 8b 50 08 <0f> b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd 48 8b 4a 20</div>
<div>[92966.374524] bckfl[29148]: segfault at f0b73934 ip 00007f4d9a95a867 sp 00007f4d981ab6c0 error 4 in select_cons_tres.so[7f4d9a93b000+2d000]</div>
<div>[92966.374538] Code: 8b 52 30 4c 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 <48> 8b 50 08 0f b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd</div>
<div>[103685.791492] sched_agent[131341]: segfault at 12 ip 00007f69d3df186b sp 00007f69d112e3b0 error 4 in select_cons_tres.so[7f69d3dd2000+2d000]</div>
<div>[103685.791505] Code: 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 48 8b 50 08 <0f> b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd 48 8b 4a 20</div>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div class="x_elementToProof" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
</div>
</body>
</html>