<meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div dir="ltr">Who owns the munge directory and key? Is it the right uid/gid? Is the munge daemon running?<div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">--<br>Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>Research Computing Services | Business Services<br>The University of Melbourne, Victoria 3010 Australia<br><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 16 Apr 2020 at 04:57, Dean Schulze <<a href="mailto:dean.w.schulze@gmail.com">dean.w.schulze@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif"><strong>UoM notice: External email. Be cautious of links, attachments, or impersonation attempts.</strong><br></div><hr></div>/etc/munge is 700<div>/etc/munge/munge.key is 400</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 15, 2020 at 12:11 PM Riebs, Andy <<a href="mailto:andy.riebs@hpe.com" target="_blank">andy.riebs@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Two trivial things to check:<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>1.<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Permissions on /etc/munge and /etc/munge.key<u></u><u></u></span></p>
<p><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><span>2.<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Is munged running on the problem node?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Andy<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11pt;font-family:Calibri,sans-serif"> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>]
<b>On Behalf Of </b>Dean Schulze<br>
<b>Sent:</b> Wednesday, April 15, 2020 1:57 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] Munge decode failing on new node<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">I've installed two new nodes onto my slurm cluster. One node works, but the other one complains about an invalid credential for munge. I've verified that the munge.key is the same as on all other nodes with<u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
sudo cksum /etc/munge/munge.key<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I recopied a munge.key from a node that works. I've verified that munge uid and gid are the same on the nodes. The time is in sync on all nodes. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Here is what is in the slurmd.log:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> error: Unable to register: Unable to contact slurm controller (connect failure)<br>
error: Munge decode failed: Invalid credential<br>
ENCODED: Wed Dec 31 17:00:00 1969<br>
DECODED: Wed Dec 31 17:00:00 1969<br>
error: authentication: Invalid authentication credential<br>
error: slurm_receive_msg_and_forward: Protocol authentication error<br>
error: service_connection: slurm_receive_msg: Protocol authentication error<br>
error: Unable to register: Unable to contact slurm controller (connect failure)<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've checked in the munged.log and all it says is <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Invalid credential <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for your help<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div>
</blockquote></div>