<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
You could enable debug logging on your slurm controllers to see if that provides some more useful info. I'd also check your firewall settings to make sure your not blocking some traffic that you shouldn't.
<code>iptables -F</code> will clear your local Linux firewall.</div>
<div class="elementToProof" style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">I'd also triple check the UID on all the systems and run this on all your compute nodes, slurm controllers, and slurmdb to make sure it
is the same! 🙂<br>
<br>
<code>id 59999</code></span></div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">I'd also restart all the slurm daemons all the systems to make sure that you don't have systems that running a daemon from before you
created UID 59999 as running processes often don't pick up changes like that unless they're restarted.</span></div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><br>
</span></div>
<div class="elementToProof"><span style="font-family: Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Cheers</span></div>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">
<div><span style="font-family: arial, sans-serif; font-size: 9.75pt; color: rgb(34, 34, 34);">-- </span><span style="font-size: 13px;"><br>
</span></div>
<div style="direction: ltr;"><span style="font-family: Helvetica; font-size: 9.75pt; color: rgb(34, 34, 34);">Mick Timony</span></div>
<div style="direction: ltr;"><span style="font-family: Helvetica; font-size: 9.75pt; color: rgb(34, 34, 34); background-color: rgb(255, 255, 255);">Senior DevOps Engineer</span><span style="font-family: Helvetica; font-size: 9.75pt; color: rgb(34, 34, 34);"><br>
Harvard Medical School</span></div>
<div style="direction: ltr;"><span style="font-family: Helvetica; font-size: 9.75pt; color: rgb(34, 34, 34);">--</span></div>
<div style="direction: ltr;"><span style="font-family: Helvetica; font-size: 9.75pt; color: rgb(34, 34, 34);"><br>
</span></div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Craig Stark <cestark@ad.uci.edu><br>
<b>Sent:</b> Monday, January 8, 2024 5:46 PM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] DBD_SEND_MULT_MSG - invalid uid error</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_PlainText x_elementToProof">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200); border-top-color:rgb(200,200,200); border-right-color:rgb(200,200,200); border-bottom-color:rgb(200,200,200); padding-left:1ex; margin-left:0.8ex; color:rgb(102,102,102)">
This ticket with SchedMD implies it's a munged issue:<br>
<br>
<a href="https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__;!!CzAuKJ42GuquVTTmVmPViYEvSg!N2M1a84yfU8mhdQ87LnBMQxye_nBsrTzTow7spIqZaQ2dLevBDZy4oNMT8KzMsmhxdRwchIht3Tgl3p8cMHhFOg9ry546OQ_iA$" data-auth="NotApplicable" id="OWAf88d737f-bdf2-09dc-7f69-b9664dd6d534" class="x_OWAAutoLink" data-loopstyle="linkonly">https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__;!!CzAuKJ42GuquVTTmVmPViYEvSg!N2M1a84yfU8mhdQ87LnBMQxye_nBsrTzTow7spIqZaQ2dLevBDZy4oNMT8KzMsmhxdRwchIht3Tgl3p8cMHhFOg9ry546OQ_iA$</a>
<br>
<br>
Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts?<br>
</blockquote>
Thanks Mick,</div>
<div class="x_PlainText x_elementToProof"><br>
</div>
<div class="x_PlainText x_elementToProof">munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the
munge.key file on the compute nodes and host node says they're the same, so I think I can put that aside.</div>
<div class="x_PlainText x_elementToProof"><br>
</div>
<div class="x_PlainText x_elementToProof">The management node runs chrony and the compute nodes sync to the management node. </div>
<div class="x_PlainText x_elementToProof x_ContentPasted0">[root@kirby uber]# chronyc tracking
<div class="x_ContentPasted0">Reference ID : 4A06A849 (t2.time.gq1.yahoo.com)</div>
<div class="x_ContentPasted0">Stratum : 3</div>
<div class="x_ContentPasted0">Ref time (UTC) : Mon Jan 08 22:26:44 2024</div>
<div class="x_ContentPasted0">System time : 0.000032525 seconds slow of NTP time</div>
<div class="x_ContentPasted0">Last offset : -0.000021390 seconds</div>
<div class="x_ContentPasted0">RMS offset : 0.000055729 seconds</div>
<div class="x_ContentPasted0">Frequency : 38.797 ppm slow</div>
<div class="x_ContentPasted0">Residual freq : +0.001 ppm</div>
<div class="x_ContentPasted0">Skew : 0.018 ppm</div>
<div class="x_ContentPasted0">Root delay : 0.033342984 seconds</div>
<div class="x_ContentPasted0">Root dispersion : 0.000524800 seconds</div>
<div class="x_ContentPasted0">Update interval : 256.8 seconds</div>
Leap status : Normal<br>
</div>
<div class="x_PlainText x_elementToProof x_ContentPasted0"><br>
</div>
<div class="x_PlainText x_elementToProof x_ContentPasted0">vs</div>
<div class="x_PlainText x_elementToProof x_ContentPasted0 x_ContentPasted1">[root@sonic01 ~]# chronyc tracking
<div class="x_ContentPasted1">Reference ID : C0A80102 (warewulf)</div>
<div class="x_ContentPasted1">Stratum : 4</div>
<div class="x_ContentPasted1">Ref time (UTC) : Mon Jan 08 22:31:02 2024</div>
<div class="x_ContentPasted1">System time : 0.000000120 seconds slow of NTP time</div>
<div class="x_ContentPasted1">Last offset : -0.000000092 seconds</div>
<div class="x_ContentPasted1">RMS offset : 0.000014737 seconds</div>
<div class="x_ContentPasted1">Frequency : 47.495 ppm slow</div>
<div class="x_ContentPasted1">Residual freq : +0.000 ppm</div>
<div class="x_ContentPasted1">Skew : 0.066 ppm</div>
<div class="x_ContentPasted1">Root delay : 0.033458963 seconds</div>
<div class="x_ContentPasted1">Root dispersion : 0.000283949 seconds</div>
<div class="x_ContentPasted1">Update interval : 64.2 seconds</div>
Leap status : Normal<br>
</div>
<div class="x_PlainText x_elementToProof"><br>
</div>
<div class="x_PlainText x_elementToProof">So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes</div>
</span></font></div>
</div>
</body>
</html>