<div dir="ltr">Hello Michael,<div><br></div><div>Thank you so much for your response. I have taken my time to explore the problem. The network seems stable, all nodes are connected to the same network where three are working but this 104 is down. 104 system has the same hardware specifications as all the others. Installations were done side by side on every node. I have tried restarting slurm several times but the node goes from idle* to down*. The details of some commands are given below.<br><br><br><b>[root@104 ~]# systemctl status slurmd.service</b><br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Sat 2022-12-03 22:50:00 PKT; 11min ago<br> Main PID: 18754 (slurmd)<br> Tasks: 1<br> Memory: 672.0K<br> CGroup: /system.slice/slurmd.service<br> └─18754 /usr/sbin/slurmd -D -s<br><br>Dec 03 22:50:00 104 systemd[1]: Started Slurm node daemon.<br>Dec 03 22:50:00 104 slurmd[18754]: slurmd: slurmd version 21.08.4 started<br>Dec 03 22:50:00 104 slurmd[18754]: slurmd: killing old slurmd[18744]<br><br><b>[root@nousheen ~]# ping 192.168.60.104</b><br>PING 192.168.60.104 (192.168.60.104) 56(84) bytes of data.<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=1 ttl=64 time=0.284 ms<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=2 ttl=64 time=0.254 ms<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=3 ttl=64 time=0.269 ms<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=4 ttl=64 time=0.260 ms<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=5 ttl=64 time=0.275 ms<br>64 bytes from <a href="http://192.168.60.104">192.168.60.104</a>: icmp_seq=6 ttl=64 time=0.262 ms<br><br><b>[root@nousheen ~]# vi /var/log/slurmctld.log</b><br>[2022-12-02T20:03:14.428] error: Nodes 104 not responding<br>[2022-12-02T20:04:58.686] error: Nodes 104 not responding, setting DOWN<br>[2022-12-02T20:54:09.863] Node 104 now responding<br>[2022-12-02T20:54:09.863] node 104 returned to service<br>[2022-12-02T20:58:14.691] error: Nodes 104 not responding<br>[2022-12-02T21:01:38.007] error: Nodes 104 not responding, setting DOWN<br>[2022-12-03T13:58:40.878] update_node: node 104 state set to IDLE<br>[2022-12-03T14:03:14.142] error: Nodes 104 not responding<br>[2022-12-03T14:05:20.392] error: Nodes 104 not responding, setting DOWN<br>[2022-12-03T22:02:40.680] _job_complete: JobId=120 WEXITSTATUS 0<br>[2022-12-03T22:02:40.680] _job_complete: JobId=120 done<br>[2022-12-03T22:33:36.036] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=121 uid 1000<br>[2022-12-03T22:33:36.064] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=122 uid 1000<br>[2022-12-03T22:33:36.118] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=123 uid 1000<br>[2022-12-03T22:43:18.943] update_node: node 104 state set to IDLE<br>[2022-12-03T22:48:14.660] error: Nodes 104 not responding<br>[2022-12-03T22:48:15.257] Node 104 now responding<br>[2022-12-03T22:53:14.142] error: Nodes 104 not responding<br>[2022-12-03T22:54:59.397] error: Nodes 104 not responding, setting DOWN<br>[2022-12-03T23:15:30.236] _slurm_rpc_submit_batch_job: JobId=124 InitPrio=4294901748 usec=588<br>[2022-12-03T23:15:30.274] sched/backfill: _start_job: Started JobId=124 in debug on 101<br><br><br><b>[root@104 ~]# vi /var/log/slurmd.log</b><br>[2022-12-03T22:48:18.255] debug: jobacct_gather/none: init: Job accounting gather NOT_INVOKED plugin loaded<br>[2022-12-03T22:48:18.255] debug: job_container/none: init: job_container none plugin loaded<br>[2022-12-03T22:48:18.256] debug: switch Cray/Aries plugin loaded.<br>[2022-12-03T22:48:18.256] debug: switch/none: init: switch NONE plugin loaded<br>[2022-12-03T22:48:18.257] slurmd started on Sat, 03 Dec 2022 22:48:18 +0500<br>[2022-12-03T22:48:18.262] CPUs=12 Boards=1 Sockets=1 Cores=6 Threads=2 Memory=31889 TmpDisk=106648 Uptime=819160 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)<br>[2022-12-03T22:48:18.262] debug: acct_gather_energy/none: init: AcctGatherEnergy NONE plugin loaded<br>[2022-12-03T22:48:18.263] debug: acct_gather_Profile/none: init: AcctGatherProfile NONE plugin loaded<br>[2022-12-03T22:48:18.263] debug: acct_gather_interconnect/none: init: AcctGatherInterconnect NONE plugin loaded<br>[2022-12-03T22:48:18.263] debug: acct_gather_filesystem/none: init: AcctGatherFilesystem NONE plugin loaded<br>[2022-12-03T22:48:18.263] debug2: No acct_gather.conf file (/etc/slurm/acct_gather.conf)<br>[2022-12-03T22:48:18.265] debug: _handle_node_reg_resp: slurmctld sent back 8 TRES.<br>[2022-12-03T22:50:00.284] slurmd version 21.08.4 started<br>[2022-12-03T22:50:00.284] killing old slurmd[18744]<br><br>currently, only one job is running on 101 which shows mix state. I don't understand why is 101 in mix state when the only thing running on it is one job. secondly, what does asterisk means for 104 (down*)? <br><br>PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>debug* up infinite 1 down* 104<br>debug* up infinite 1 mix 101<br>debug* up infinite 2 idle 105,nousheen<br><br><br>The output given in slurmd.log is beyond my understanding. I have searched for it on the internet but failed to find a proper solution. Any kind of help in understanding the problem would be highly appreciated. Thank you for your time.<br><br>Best regards,<br><br>Nousheen <br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div></div></div></div></div></div><br></div></div><div hspace="streak-pt-mark" style="max-height:1px"><img alt="" style="width:0px;max-height:0px;overflow:hidden" src="https://mailfoogae.appspot.com/t?sender=abm91c2hlZW5wYXJ2YWl6QGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=6f03c110-f531-40df-ad24-0a537d4f865a"><font color="#ffffff" size="1">ᐧ</font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 2, 2022 at 10:42 PM Michael Robbert <<a href="mailto:mrobbert@mines.edu">mrobbert@mines.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg4116178717823841512"><div lang="EN-US" style="overflow-wrap: break-word;"><div class="m_4116178717823841512WordSection1"><p class="MsoNormal"><span style="font-size:11pt">Nousheen,<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt">When a node is not responding the first place to start is to ensure that the node is up and slurmd is running. It looks like you have confirmed that with your output from the command “scontrol show slurmd” so that is a good start. After verifying that slurmd is running the next step would be to examine the logs. Look at /var/log/slurmd.log on that node and see if it tells you why it can’t communicate with the slurm controller. <u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt">Other things to think about since this is a new setup are to make sure the network is stable and that DNS is working properly for all nodes. Make sure that all nodes in the cluster can do correct DNS resolution of all other nodes in the cluster.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p><div><div><p class="MsoNormal"><b><span style="font-size:11pt;color:rgb(0,32,96)">Mike Robbert</span></b><span style="font-size:11pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><b><span style="font-size:11pt;color:rgb(0,32,96)">Cyberinfrastructure Specialist, Cyberinfrastructure and Advanced Research Computing</span></b><span style="font-size:11pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;color:rgb(118,113,113)">Information and Technology Solutions (ITS)</span><span style="font-size:11pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;color:rgb(118,113,113)">303-273-3786 | </span><span style="font-size:11pt;color:black"><a href="mailto:mrobbert@mines.edu" target="_blank"><span style="color:rgb(5,99,193)">mrobbert@mines.edu</span></a></span><span style="font-size:11pt;color:rgb(118,113,113)"> </span><span style="font-size:12pt;color:rgb(118,113,113)"> </span><span style="font-size:11pt;color:black"><u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:11pt;color:black"><img border="0" width="208" height="38" style="width: 2.1666in; height: 0.3958in;" id="m_4116178717823841512Picture_x0020_1" src="cid:184d9764d7a4cff311" alt="A close up of a sign
Description automatically generated"><u></u><u></u></span></p><p class="MsoNormal"><b><span style="font-size:11pt;color:rgb(43,65,96)">Our values:</span></b><span style="font-size:11pt;color:rgb(43,65,96)"> </span><span style="font-size:11pt;color:rgb(118,113,113)">Trust | Integrity | Respect | Responsibility</span><span style="font-size:11pt;color:black"><u></u><u></u></span></p></div></div><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p><div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0in 0in"><p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Nousheen <<a href="mailto:nousheenparvaiz@gmail.com" target="_blank">nousheenparvaiz@gmail.com</a>><br><b>Date: </b>Friday, December 2, 2022 at 09:22<br><b>To: </b>Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br><b>Subject: </b>Re: [slurm-users] [External] ERROR: slurmctld: auth/munge: _print_cred: DECODED<u></u><u></u></span></p></div><div style="border:1pt solid rgb(156,101,0);padding:2pt"><p class="MsoNormal" style="line-height:12pt;background:rgb(255,235,156)"><b><span style="color:rgb(156,101,0)">CAUTION:</span></b><span style="color:black"> This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.<u></u><u></u></span></p></div><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p><div><div><p class="MsoNormal"><span style="font-size:11pt"><br clear="all"><u></u><u></u></span></p><div><div><div><div><div><div><p class="MsoNormal"><span style="font-size:11pt">Dear Ole,<br><br>Thank you so much for your response. I have now adjusted the RealMemory in the slurm.conf which was set by default previously. Your insight was really helpful. Now, when I submit the job, it is running on three nodes but one node (104) is not responding. The details of some commands are given below.<br><br><br><b>[root@nousheen ~]# squeue -j</b><br> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br> 120 debug SRBD-1 nousheen R 0:54 1 101<br> 121 debug SRBD-2 nousheen R 0:54 1 105<br> 122 debug SRBD-3 nousheen R 0:54 1 nousheen<br> 123 debug SRBD-4 nousheen R 0:54 2 105,nousheen<br> <br> <b><br>[root@nousheen ~]# scontrol show nodes</b><br>NodeName=101 Arch=x86_64 CoresPerSocket=6 <br> CPUAlloc=8 CPUTot=12 CPULoad=0.01<br> AvailableFeatures=(null)<br> ActiveFeatures=(null)<br> Gres=(null)<br> NodeAddr=192.168.60.118 NodeHostName=101 Version=21.08.4<br> OS=Linux 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 <br> RealMemory=31919 AllocMem=0 FreeMem=293 Sockets=1 Boards=1<br> State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br> Partitions=debug <br> BootTime=2022-11-24T11:18:28 SlurmdStartTime=2022-12-02T19:56:01<br> LastBusyTime=2022-12-02T19:58:14<br> CfgTRES=cpu=12,mem=31919M,billing=12<br> AllocTRES=cpu=8<br> CapWatts=n/a<br> CurrentWatts=0 AveWatts=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br><br>NodeName=104 Arch=x86_64 CoresPerSocket=6 <br> CPUAlloc=0 CPUTot=12 CPULoad=0.01<br> AvailableFeatures=(null)<br> ActiveFeatures=(null)<br> Gres=(null)<br> NodeAddr=192.168.60.104 NodeHostName=104 Version=21.08.4<br> OS=Linux 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 <br> RealMemory=31889 AllocMem=0 FreeMem=30433 Sockets=1 Boards=1<br> State=IDLE+NOT_RESPONDING ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br> Partitions=debug <br> BootTime=2022-11-24T11:15:43 SlurmdStartTime=2022-12-02T19:57:29<br> LastBusyTime=2022-12-02T19:58:14<br> CfgTRES=cpu=12,mem=31889M,billing=12<br> AllocTRES=<br> CapWatts=n/a<br> CurrentWatts=0 AveWatts=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br><br>NodeName=105 Arch=x86_64 CoresPerSocket=6 <br> CPUAlloc=12 CPUTot=12 CPULoad=1.03<br> AvailableFeatures=(null)<br> ActiveFeatures=(null)<br> Gres=(null)<br> NodeAddr=192.168.60.105 NodeHostName=105 Version=21.08.4<br> OS=Linux 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 <br> RealMemory=32051 AllocMem=0 FreeMem=14874 Sockets=1 Boards=1<br> State=ALLOCATED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br> Partitions=debug <br> BootTime=2022-11-24T11:15:37 SlurmdStartTime=2022-12-02T19:56:57<br> LastBusyTime=2022-12-02T19:58:14<br> CfgTRES=cpu=12,mem=32051M,billing=12<br> AllocTRES=cpu=12<br> CapWatts=n/a<br> CurrentWatts=0 AveWatts=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br><br>NodeName=nousheen Arch=x86_64 CoresPerSocket=6 <br> CPUAlloc=12 CPUTot=12 CPULoad=0.32<br> AvailableFeatures=(null)<br> ActiveFeatures=(null)<br> Gres=(null)<br> NodeAddr=192.168.60.194 NodeHostName=nousheen Version=21.08.5<br> OS=Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 <br> RealMemory=31889 AllocMem=0 FreeMem=16666 Sockets=1 Boards=1<br> State=ALLOCATED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br> Partitions=debug <br> BootTime=2022-12-01T12:00:18 SlurmdStartTime=2022-12-02T19:56:36<br> LastBusyTime=2022-12-02T19:58:15<br> CfgTRES=cpu=12,mem=31889M,billing=12<br> AllocTRES=cpu=12<br> CapWatts=n/a<br> CurrentWatts=0 AveWatts=0<br> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br><br><br><b>[root@104 ~]# scontrol show slurmd</b><br>Active Steps = NONE<br>Actual CPUs = 12<br>Actual Boards = 1<br>Actual sockets = 1<br>Actual cores = 6<br>Actual threads per core = 2<br>Actual real memory = 31889 MB<br>Actual temp disk space = 106648 MB<br>Boot time = 2022-12-02T19:57:29<br>Hostname = 104<br>Last slurmctld msg time = NONE<br>Slurmd PID = 16906<br>Slurmd Debug = 3<br>Slurmd Logfile = /var/log/slurmd.log<br>Version = 21.08.4<br><br><br>If you can give me a hint to as what can be the reason behind one node nonresponding or what files or problems I should focus on, I would be highly grateful to you. Thank you for your time.<br><br>Best regards,<br><br>Nousheen <u></u><u></u></span></p></div></div></div></div></div></div><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p></div><div><p class="MsoNormal"><span style="font-size:11pt"><img border="0" id="m_4116178717823841512_x0000_i1025" src="https://mailfoogae.appspot.com/t?sender=abm91c2hlZW5wYXJ2YWl6QGdtYWlsLmNvbQ%3D%3D&type=zerocontent&guid=62ff6c27-71c9-4b7c-958e-2800dee489f1"></span><span style="font-size:7.5pt;font-family:"Euphemia UCAS",sans-serif;color:white">ᐧ</span><span style="font-size:11pt"><u></u><u></u></span></p></div><p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p><div><div><p class="MsoNormal"><span style="font-size:11pt">On Fri, Dec 2, 2022 at 11:56 AM Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" target="_blank">Ole.H.Nielsen@fysik.dtu.dk</a>> wrote:<u></u><u></u></span></p></div><blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:11pt">Hi Nousheen,<br><br>It seems that you have configured incorrectly the nodes in slurm.conf. I <br>notice this:<br><br> RealMemory=1<br><br>This means 1 Megabyte of RAM memory, we only had this with IBM PCs back in <br>the 1980ies :-)<br><br>See how to configure nodes in <br><a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysik.dtu.dk%2FNiflheim_system%2FSlurm_configuration%2F%23compute-node-configuration&data=05%7C01%7Cmrobbert%40mines.edu%7C76f719a37bbf421170de08dad4817173%7C997209e009b346239a4d76afa44a675c%7C0%7C0%7C638055949673005416%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YpIYqfpJPUQgDUpxXO2GsX5kn7GpkD5DJsysANsQhmQ%3D&reserved=0" target="_blank">https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#compute-node-configuration</a><br><br>You must run "slurmd -C" on each node to determine its actual hardware.<br><br>I hope this helps.<br><br>/Ole<br><br>On 12/1/22 21:08, Nousheen wrote:<br>> Dear Robbert,<br>> <br>> Thankyou so much for your response. I was so focused on sync of time that <br>> I missed the date on one of the nodes which was 1 day behind as you said. <br>> I have corrected it and now i get the following output in status.<br>> <br>> *(base) [nousheen@nousheen slurm]$ systemctl status slurmctld.service -l*<br>> ● slurmctld.service - Slurm controller daemon<br>> Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled; vendor <br>> preset: disabled)<br>> Active: active (running) since Thu 2022-12-01 21:37:34 PKT; 20min ago<br>> Main PID: 19475 (slurmctld)<br>> Tasks: 10<br>> Memory: 4.5M<br>> CGroup: /system.slice/slurmctld.service<br>> </span><span style="font-size:11pt;font-family:"MS Gothic"">├</span><span style="font-size:11pt">─19475 /usr/sbin/slurmctld -D -s<br>> └─19538 slurmctld: slurmscriptd<br>> <br>> Dec 01 21:47:08 nousheen slurmctld[19475]: slurmctld: sched/backfill: <br>> _start_job: Started JobId=106 in debug on 101<br>> Dec 01 21:47:09 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=106 WEXITSTATUS 1<br>> Dec 01 21:47:09 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=106 done<br>> Dec 01 21:47:11 nousheen slurmctld[19475]: slurmctld: sched: Allocate <br>> JobId=107 NodeList=101 #CPUs=8 Partition=debug<br>> Dec 01 21:47:11 nousheen slurmctld[19475]: slurmctld: sched: Allocate <br>> JobId=108 NodeList=105 #CPUs=8 Partition=debug<br>> Dec 01 21:47:11 nousheen slurmctld[19475]: slurmctld: sched: Allocate <br>> JobId=109 NodeList=nousheen #CPUs=8 Partition=debug<br>> Dec 01 21:47:11 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=107 WEXITSTATUS 1<br>> Dec 01 21:47:11 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=107 done<br>> Dec 01 21:47:12 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=108 WEXITSTATUS 1<br>> Dec 01 21:47:12 nousheen slurmctld[19475]: slurmctld: _job_complete: <br>> JobId=108 done<br>> <br>> I have total four nodes one of which is the server node. After submitting <br>> a job, the job only runs at my server compute node while all the other <br>> nodes are IDLE, DOWN or nonresponding. The details are given below:<br>> <br>> *(base) [nousheen@nousheen slurm]$ scontrol show nodes*<br>> NodeName=101 Arch=x86_64 CoresPerSocket=6<br>> CPUAlloc=0 CPUTot=12 CPULoad=0.01<br>> AvailableFeatures=(null)<br>> ActiveFeatures=(null)<br>> Gres=(null)<br>> NodeAddr=192.168.60.101 NodeHostName=101 Version=21.08.4<br>> OS=Linux 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022<br>> RealMemory=1 AllocMem=0 FreeMem=641 Sockets=1 Boards=1<br>> State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br>> Partitions=debug<br>> BootTime=2022-11-24T11:18:28 SlurmdStartTime=2022-12-01T21:34:57<br>> LastBusyTime=2022-12-02T00:58:31<br>> CfgTRES=cpu=12,mem=1M,billing=12<br>> AllocTRES=<br>> CapWatts=n/a<br>> CurrentWatts=0 AveWatts=0<br>> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>> <br>> NodeName=104 CoresPerSocket=6<br>> CPUAlloc=0 CPUTot=12 CPULoad=N/A<br>> AvailableFeatures=(null)<br>> ActiveFeatures=(null)<br>> Gres=(null)<br>> NodeAddr=192.168.60.114 NodeHostName=104<br>> RealMemory=1 AllocMem=0 FreeMem=N/A Sockets=1 Boards=1<br>> State=DOWN+NOT_RESPONDING ThreadsPerCore=2 TmpDisk=0 Weight=1 <br>> Owner=N/A MCS_label=N/A<br>> Partitions=debug<br>> BootTime=None SlurmdStartTime=None<br>> LastBusyTime=2022-12-01T21:37:35<br>> CfgTRES=cpu=12,mem=1M,billing=12<br>> AllocTRES=<br>> CapWatts=n/a<br>> CurrentWatts=0 AveWatts=0<br>> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>> Reason=Not responding [slurm@2022-12-01T16:22:28]<br>> <br>> NodeName=105 Arch=x86_64 CoresPerSocket=6<br>> CPUAlloc=0 CPUTot=12 CPULoad=1.08<br>> AvailableFeatures=(null)<br>> ActiveFeatures=(null)<br>> Gres=(null)<br>> NodeAddr=192.168.60.115 NodeHostName=105 Version=21.08.4<br>> OS=Linux 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022<br>> RealMemory=1 AllocMem=0 FreeMem=20723 Sockets=1 Boards=1<br>> State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br>> Partitions=debug<br>> BootTime=2022-11-24T11:15:37 SlurmdStartTime=2022-12-01T16:15:30<br>> LastBusyTime=2022-12-01T21:47:11<br>> CfgTRES=cpu=12,mem=1M,billing=12<br>> AllocTRES=<br>> CapWatts=n/a<br>> CurrentWatts=0 AveWatts=0<br>> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>> <br>> NodeName=nousheen Arch=x86_64 CoresPerSocket=6<br>> CPUAlloc=8 CPUTot=12 CPULoad=6.73<br>> AvailableFeatures=(null)<br>> ActiveFeatures=(null)<br>> Gres=(null)<br>> NodeAddr=192.168.60.149 NodeHostName=nousheen Version=21.08.5<br>> OS=Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021<br>> RealMemory=1 AllocMem=0 FreeMem=22736 Sockets=1 Boards=1<br>> State=MIXED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A<br>> Partitions=debug<br>> BootTime=2022-12-01T12:00:08 SlurmdStartTime=2022-12-01T12:00:42<br>> LastBusyTime=2022-12-01T21:37:39<br>> CfgTRES=cpu=12,mem=1M,billing=12<br>> AllocTRES=cpu=8<br>> CapWatts=n/a<br>> CurrentWatts=0 AveWatts=0<br>> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>> <br>> Where as this command shows only one node on which job is running:<br>> <br>> *(base) [nousheen@nousheen slurm]$ squeue -j*<br>> JOBID PARTITION NAME USER ST TIME NODES <br>> NODELIST(REASON)<br>> 109 debug SRBD-4 nousheen R 3:17:48 1 nousheen<br>> <br>> Can you please guide me as to why my compute nodes are down and not working?<br>> <br>> Thank you for your time.<br>> <br>> <br>> Best Regards,<br>> Nousheen Parvaiz<br>> <br>> <br>> </span><span style="font-size:11pt;font-family:"Euphemia UCAS",sans-serif">ᐧ</span><span style="font-size:11pt"><br>> <br>> On Thu, Dec 1, 2022 at 8:55 PM Michael Robbert <<a href="mailto:mrobbert@mines.edu" target="_blank">mrobbert@mines.edu</a> <br>> <mailto:<a href="mailto:mrobbert@mines.edu" target="_blank">mrobbert@mines.edu</a>>> wrote:<br>> <br>> I believe that the error you need to pay attention to for this issue<br>> is this line:____<br>> <br>> __ __<br>> <br>> Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: error: Check for<br>> out of sync clocks____<br>> <br>> __ __<br>> <br>> __ __<br>> <br>> It looks like your compute nodes clock is a full day ahead of your<br>> controller node. Dec. 2 instead of Dec. 1. The clocks need to be in<br>> sync for munge to work.____<br>> <br>> __ __<br>> <br>> *Mike Robbert*____<br>> <br>> *Cyberinfrastructure Specialist, Cyberinfrastructure and Advanced<br>> Research Computing*____<br>> <br>> Information and Technology Solutions (ITS)____<br>> <br>> 303-273-3786 | <a href="mailto:mrobbert@mines.edu" target="_blank">mrobbert@mines.edu</a> <mailto:<a href="mailto:mrobbert@mines.edu" target="_blank">mrobbert@mines.edu</a>>____<br>> <br>> A close up of a sign Description automatically generated____<br>> <br>> *Our values:*Trust | Integrity | Respect | Responsibility____<br>> <br>> __ __<br>> <br>> __ __<br>> <br>> *From: *slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a><br>> <mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>> on behalf of Nousheen<br>> <<a href="mailto:nousheenparvaiz@gmail.com" target="_blank">nousheenparvaiz@gmail.com</a> <mailto:<a href="mailto:nousheenparvaiz@gmail.com" target="_blank">nousheenparvaiz@gmail.com</a>>><br>> *Date: *Thursday, December 1, 2022 at 06:19<br>> *To: *Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>> <mailto:<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>>><br>> *Subject: *[External] [slurm-users] ERROR: slurmctld: auth/munge:<br>> _print_cred: DECODED____<br>> <br>> *CAUTION:*This email originated from outside of the Colorado School of<br>> Mines organization. Do not click on links or open attachments unless<br>> you recognize the sender and know the content is safe.____<br>> <br>> __ __<br>> <br>> __ __<br>> <br>> __ __<br>> <br>> Hello Everyone,____<br>> <br>> __ __<br>> <br>> I am using slurm version 21.08.5 and Centos 7.____<br>> <br>> __ __<br>> <br>> I successfully start slurmd on all compute nodes but when I start<br>> slurmctld on server node it gives the following error:____<br>> <br>> __ __<br>> <br>> *(base) [nousheen@nousheen ~]$ systemctl status slurmctld.service -l*<br>> ● slurmctld.service - Slurm controller daemon<br>> Loaded: loaded (/etc/systemd/system/slurmctld.service; enabled;<br>> vendor preset: disabled)<br>> Active: active (running) since Thu 2022-12-01 12:00:42 PKT; 4h<br>> 16min ago<br>> Main PID: 1631 (slurmctld)<br>> Tasks: 10<br>> Memory: 4.0M<br>> CGroup: /system.slice/slurmctld.service<br>> </span><span style="font-size:11pt;font-family:"MS Gothic"">├</span><span style="font-size:11pt">─1631 /usr/sbin/slurmctld -D -s<br>> └─1818 slurmctld: slurmscriptd<br>> <br>> Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: auth/munge:<br>> _print_cred: DECODED: Thu Dec 01 16:17:19 2022<br>> Dec 01 16:17:19 nousheen slurmctld[1631]: slurmctld: error: Check for<br>> out of sync clocks<br>> Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: error: Munge<br>> decode failed: Rewound credential<br>> Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: auth/munge:<br>> _print_cred: ENCODED: Fri Dec 02 16:16:55 2022<br>> Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: auth/munge:<br>> _print_cred: DECODED: Thu Dec 01 16:17:20 2022<br>> Dec 01 16:17:20 nousheen slurmctld[1631]: slurmctld: error: Check for<br>> out of sync clocks<br>> Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: error: Munge<br>> decode failed: Rewound credential<br>> Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: auth/munge:<br>> _print_cred: ENCODED: Fri Dec 02 16:16:56 2022<br>> Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: auth/munge:<br>> _print_cred: DECODED: Thu Dec 01 16:17:21 2022<br>> Dec 01 16:17:21 nousheen slurmctld[1631]: slurmctld: error: Check for<br>> out of sync clocks____<br>> <br>> __ __<br>> <br>> When I run the following command on compute nodes I get the following<br>> output:____<br>> <br>> __ __<br>> <br>> [gpu101@101 ~]$*munge -n | unmunge*____<br>> <br>> STATUS: Success (0)<br>> ENCODE_HOST: ??? (0.0.0.101)<br>> ENCODE_TIME: 2022-12-02 16:33:38 +0500 (1669980818)<br>> DECODE_TIME: 2022-12-02 16:33:38 +0500 (1669980818)<br>> TTL: 300<br>> CIPHER: aes128 (4)<br>> MAC: sha1 (3)<br>> ZIP: none (0)<br>> UID: gpu101 (1000)<br>> GID: gpu101 (1000)<br>> LENGTH: 0____<br>> <br>> __ __<br>> <br>> Is this error because the encode_host name has question marks and the<br>> IP is also not picked correctly by munge. How can I correct this? All<br>> the nodes keep non-responding when I run a job. However, I have all<br>> the clocks synced across the cluster. ____<br>> <br>> __ __<br>> <br>> I am new to slurm. Kindly guide me in this matter.____<u></u><u></u></span></p></blockquote></div></div></div></div></div></blockquote></div>