<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Euphemia UCAS";
panose-1:2 11 5 3 4 1 2 2 1 4;}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>I believe that the error you need to pay attention to for this issue is this line:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:13.5pt;color:black'>Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: error: Check for out of sync clocks</span><span style='font-size:11.0pt'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>It looks like your compute nodes clock is a full day ahead of your controller node. Dec. 2 instead of Dec. 1. The clocks need to be in sync for munge to work.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div><div><p class=MsoNormal><b><span style='font-size:11.0pt;color:#002060'>Mike Robbert</span></b><span style='font-size:11.0pt;color:black'><o:p></o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;color:#002060'>Cyberinfrastructure Specialist, Cyberinfrastructure and Advanced Research Computing</span></b><span style='font-size:11.0pt;color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;color:#767171'>Information and Technology Solutions (ITS)</span><span style='font-size:11.0pt;color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;color:#767171'>303-273-3786 | </span><span style='font-size:11.0pt;color:black'><a href="mailto:mrobbert@mines.edu"><span style='color:#0563C1'>mrobbert@mines.edu</span></a></span><span style='font-size:11.0pt;color:#767171'> </span><span style='font-size:12.0pt;color:#767171'> </span><span style='font-size:11.0pt;color:black'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;color:black'><img border=0 width=208 height=38 style='width:2.1666in;height:.3958in' id="Picture_x0020_1" src="cid:image001.png@01D90562.47CF1D70" alt="A close up of a sign
Description automatically generated"><o:p></o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;color:#2B4160'>Our values:</span></b><span style='font-size:11.0pt;color:#2B4160'> </span><span style='font-size:11.0pt;color:#767171'>Trust | Integrity | Respect | Responsibility</span><span style='font-size:11.0pt;color:black'><o:p></o:p></span></p></div></div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Nousheen <nousheenparvaiz@gmail.com><br><b>Date: </b>Thursday, December 1, 2022 at 06:19<br><b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br><b>Subject: </b>[External] [slurm-users] ERROR: slurmctld: auth/munge: _print_cred: DECODED<o:p></o:p></span></p></div><div style='border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt'><p class=MsoNormal style='line-height:12.0pt;background:#FFEB9C'><b><span style='color:#9C6500'>CAUTION:</span></b><span style='color:black'> This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div><div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>Hello Everyone,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>I am using slurm version 21.08.5 and Centos 7.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'> I successfully start slurmd on all compute nodes but when I start slurmctld on server node it gives the following error:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><b><span style='font-size:11.0pt'>(base) [nousheen@nousheen ~]$ systemctl status slurmctld.service -l</span></b><span style='font-size:11.0pt'><br>● slurmctld.service - Slurm controller daemon<br> Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor preset: disabled)<br> Active: active (running) since Thu 2022-12-01 12:00:42 PKT; 4h 16min ago<br> Main PID: 1631 (slurmctld)<br> Tasks: 10<br> Memory: 4.0M<br> CGroup: /system.slice/slurmctld.service<br> </span><span style='font-size:11.0pt;font-family:"MS Gothic"'>├</span><span style='font-size:11.0pt'>─1631 /usr/sbin/slurmctld -D -s<br> └─1818 slurmctld: slurmscriptd <br><br>Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: auth/munge: _print_cred: DECODED: Thu Dec 01 16:17:19 2022<br>Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: error: Check for out of sync clocks<br>Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: error: Munge decode failed: Rewound credential<br>Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: auth/munge: _print_cred: ENCODED: Fri Dec 02 16:16:55 2022<br>Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: auth/munge: _print_cred: DECODED: Thu Dec 01 16:17:20 2022<br>Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: error: Check for out of sync clocks<br>Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: error: Munge decode failed: Rewound credential<br>Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: auth/munge: _print_cred: ENCODED: Fri Dec 02 16:16:56 2022<br>Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: auth/munge: _print_cred: DECODED: Thu Dec 01 16:17:21 2022<br>Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: error: Check for out of sync clocks<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>When I run the following command on compute nodes I get the following output:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'> [gpu101@101 ~]$<b> munge -n | unmunge</b><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt'>STATUS: Success (0)<br>ENCODE_HOST: ??? (0.0.0.101)<br>ENCODE_TIME: 2022-12-02 16:33:38 +0500 (1669980818)<br>DECODE_TIME: 2022-12-02 16:33:38 +0500 (1669980818)<br>TTL: 300<br>CIPHER: aes128 (4)<br>MAC: sha1 (3)<br>ZIP: none (0)<br>UID: gpu101 (1000)<br>GID: gpu101 (1000)<br>LENGTH: 0<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>Is this error because the encode_host name has question marks and the IP is also not picked correctly by munge. How can I correct this? All the nodes keep non-responding when I run a job. However, I have all the clocks synced across the cluster. <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>I am new to slurm. Kindly guide me in this matter.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'> <o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt'><br clear=all><o:p></o:p></span></p><div><div><div><div><div><div><p class=MsoNormal><span style='font-size:11.0pt'>Best Regards,<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Arial",sans-serif'>Nousheen Parvaiz<br>Ph.D. Scholar</span><span style='font-size:11.0pt'> <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Arial",sans-serif'><o:p> </o:p></span></p></div></div></div></div></div></div></div></div><div><p class=MsoNormal><span style='font-size:11.0pt'><img border=0 id="_x0000_i1025" src="https://mailfoogae.appspot.com/t?sender=abm91c2hlZW5wYXJ2YWl6QGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=c746fbb9-88d5-4ea9-8121-3f7f2cff3fcb"></span><span style='font-size:7.5pt;font-family:"Euphemia UCAS",sans-serif;color:white'>ᐧ</span><span style='font-size:11.0pt'><o:p></o:p></span></p></div></div></div></body></html>