<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi all</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
We installed Slurm 22.05 yesterday, and the slurmctld deamon crashes randomly every couply of hours. It's not much information I can get out of it, but running slurmctld in foreground (slurmctld -D -vvvvv) does not reveal anything before the crash.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Possibly unrelated errors reported by slurmctld:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] We need to forward this to other nodes use slurm_receive_msg_and_forward instead
<div>slurmctld: error: auth_g_unpack: authentication plugin unknown(1297436231) not found</div>
<div>slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] auth_g_unpack: 0 has authentication error: No error</div>
<div>slurmctld: error: slurm_unpack_received_msg: [[localhost]:43924] Header lengths are longer than data received</div>
<div>slurmctld: error: slurm_receive_msg [127.0.0.1:43924]: Unspecified error</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
dmesg shows:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
[87848.891824] sched_agent[163673]: segfault at 12 ip 00007fabd0f1986b sp 00007fabca1843b0 error 4
<div>[87848.891827] Code: 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 48 8b 50 08 <0f> b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd 48 8b 4a 20</div>
<div>[92966.374524] bckfl[29148]: segfault at f0b73934 ip 00007f4d9a95a867 sp 00007f4d981ab6c0 error 4 in select_cons_tres.so[7f4d9a93b000+2d000]</div>
<div>[92966.374538] Code: 8b 52 30 4c 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 <48> 8b 50 08 0f b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd</div>
<div>[103685.791492] sched_agent[131341]: segfault at 12 ip 00007f69d3df186b sp 00007f69d112e3b0 error 4 in select_cons_tres.so[7f69d3dd2000+2d000]</div>
<div>[103685.791505] Code: 89 f9 48 85 d2 74 07 44 89 e9 48 0f af ca 0f b7 c0 0f af c1 39 c3 0f 4c d8 4c 89 f7 e8 9e 5c fe ff 48 85 c0 74 31 48 8b 50 08 <0f> b7 42 12 66 85 c0 75 09 0f b7 42 42 66 85 c0 74 dd 48 8b 4a 20</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
</body>
</html>