<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText elementToProof">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
This ticket with SchedMD implies it's a munged issue:<br>
<br>
<a href="https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__;!!CzAuKJ42GuquVTTmVmPViYEvSg!N2M1a84yfU8mhdQ87LnBMQxye_nBsrTzTow7spIqZaQ2dLevBDZy4oNMT8KzMsmhxdRwchIht3Tgl3p8cMHhFOg9ry546OQ_iA$" data-auth="NotApplicable" id="OWAf88d737f-bdf2-09dc-7f69-b9664dd6d534" class="OWAAutoLink" data-loopstyle="linkonly">https://urldefense.com/v3/__https://bugs.schedmd.com/show_bug.cgi?id=1293__;!!CzAuKJ42GuquVTTmVmPViYEvSg!N2M1a84yfU8mhdQ87LnBMQxye_nBsrTzTow7spIqZaQ2dLevBDZy4oNMT8KzMsmhxdRwchIht3Tgl3p8cMHhFOg9ry546OQ_iA$</a>
<br>
<br>
Is the munge daemon running on all systems? If it is, are all servers running a network time daemon such chronyd or ntpd and the time is in sync on all hosts?<br>
</blockquote>
Thanks Mick,</div>
<div class="PlainText elementToProof"><br>
</div>
<div class="PlainText elementToProof">munge is seemingly running on all systems (systemctl status munge). I do get a warning about the munge file changing on disk, but I'm pretty sure that's from warewulf sync'ing files every minute. A sha256sum on the munge.key
file on the compute nodes and host node says they're the same, so I think I can put that aside.</div>
<div class="PlainText elementToProof"><br>
</div>
<div class="PlainText elementToProof">The management node runs chrony and the compute nodes sync to the management node. </div>
<div class="PlainText elementToProof ContentPasted0">[root@kirby uber]# chronyc tracking
<div class="ContentPasted0">Reference ID : 4A06A849 (t2.time.gq1.yahoo.com)</div>
<div class="ContentPasted0">Stratum : 3</div>
<div class="ContentPasted0">Ref time (UTC) : Mon Jan 08 22:26:44 2024</div>
<div class="ContentPasted0">System time : 0.000032525 seconds slow of NTP time</div>
<div class="ContentPasted0">Last offset : -0.000021390 seconds</div>
<div class="ContentPasted0">RMS offset : 0.000055729 seconds</div>
<div class="ContentPasted0">Frequency : 38.797 ppm slow</div>
<div class="ContentPasted0">Residual freq : +0.001 ppm</div>
<div class="ContentPasted0">Skew : 0.018 ppm</div>
<div class="ContentPasted0">Root delay : 0.033342984 seconds</div>
<div class="ContentPasted0">Root dispersion : 0.000524800 seconds</div>
<div class="ContentPasted0">Update interval : 256.8 seconds</div>
Leap status : Normal<br>
</div>
<div class="PlainText elementToProof ContentPasted0"><br>
</div>
<div class="PlainText elementToProof ContentPasted0">vs</div>
<div class="PlainText elementToProof ContentPasted0 ContentPasted1">[root@sonic01 ~]# chronyc tracking
<div class="ContentPasted1">Reference ID : C0A80102 (warewulf)</div>
<div class="ContentPasted1">Stratum : 4</div>
<div class="ContentPasted1">Ref time (UTC) : Mon Jan 08 22:31:02 2024</div>
<div class="ContentPasted1">System time : 0.000000120 seconds slow of NTP time</div>
<div class="ContentPasted1">Last offset : -0.000000092 seconds</div>
<div class="ContentPasted1">RMS offset : 0.000014737 seconds</div>
<div class="ContentPasted1">Frequency : 47.495 ppm slow</div>
<div class="ContentPasted1">Residual freq : +0.000 ppm</div>
<div class="ContentPasted1">Skew : 0.066 ppm</div>
<div class="ContentPasted1">Root delay : 0.033458963 seconds</div>
<div class="ContentPasted1">Root dispersion : 0.000283949 seconds</div>
<div class="ContentPasted1">Update interval : 64.2 seconds</div>
Leap status : Normal<br>
</div>
<div class="PlainText elementToProof"><br>
</div>
<div class="PlainText elementToProof">So, the compute node is talking to the host and the host is talking to generic NTP sources. "date" shows the same time on the compute nodes</div>
</span></font></div>
</body>
</html>