<div dir="ltr">Just noticed this. On the problem node the munged.log file has an entry every 1:40:<div><br></div><div>2020-04-17 15:31:02 -0600 Info: Invalid credential<br>2020-04-17 15:32:42 -0600 Info: Invalid credential<br>2020-04-17 15:34:22 -0600 Info: Invalid credential<br></div><div><br></div><div>This happens on the failed node and two other nodes that work. Two nodes that work (including the controller) don't have this message.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 17, 2020 at 2:00 PM Riebs, Andy <<a href="mailto:andy.riebs@hpe.com">andy.riebs@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_-7619039430370511186WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">A couple of quick checks to see if the problem is munge:<u></u><u></u></span></p>
<p class="gmail-m_-7619039430370511186MsoListParagraph"><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">On the problem node, try<br>
$ echo foo | munge | unmunge<u></u><u></u></span></p>
<p class="gmail-m_-7619039430370511186MsoListParagraph"><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2.<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">If (1) works, try this from the node running slurmctld to the problem node<br>
slurm-node$ echo foo | ssh node munge | unmunge<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>]
<b>On Behalf Of </b>Dean Schulze<br>
<b>Sent:</b> Friday, April 17, 2020 3:40 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> Re: [slurm-users] Munge decode failing on new node<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">There is no ntp service running on any of my nodes, and all but this one is working. I haven't heard that ntp is a requirement for slurm, just that the time be synchronized across the cluster. And it is.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Apr 15, 2020 at 12:17 PM Carlos Fenoy <<a href="mailto:minibit@gmail.com" target="_blank">minibit@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">I’d check ntp as your encoding time seems odd to me<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, 15 Apr 2020 at 19:59, Dean Schulze <<a href="mailto:dean.w.schulze@gmail.com" target="_blank">dean.w.schulze@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">I've installed two new nodes onto my slurm cluster. One node works, but the other one complains about an invalid credential for munge. I've verified that the munge.key is the same as on all other nodes with<u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
sudo cksum /etc/munge/munge.key<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I recopied a munge.key from a node that works. I've verified that munge uid and gid are the same on the nodes. The time is in sync on all nodes. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here is what is in the slurmd.log:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> error: Unable to register: Unable to contact slurm controller (connect failure)<br>
error: Munge decode failed: Invalid credential<br>
ENCODED: Wed Dec 31 17:00:00 1969<br>
DECODED: Wed Dec 31 17:00:00 1969<br>
error: authentication: Invalid authentication credential<br>
error: slurm_receive_msg_and_forward: Protocol authentication error<br>
error: service_connection: slurm_receive_msg: Protocol authentication error<br>
error: Unable to register: Unable to contact slurm controller (connect failure)<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've checked in the munged.log and all it says is <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Invalid credential <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for your help<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">--<br>
Carles Fenoy<u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>