<div dir="ltr"><div dir="ltr"><div>Thanks Glenn!</div><div><br></div><div>I change the slurm.conf logging to "debug5" on both the server and the client.</div><div><br></div><div>I also created /var/log/slurm on both the client and server and chown-ed to slurm:slurm.</div><div><br></div><div>On the server I did "scontrol reconfigure".</div><div><br></div><div>Then I rebooted the compute node. When I logged in, slurm was not up. I ran systemctl start slurmd. It stayed for about 5 seconds then stoppped.</div><div><br></div><div><br></div><div># systemctl status slurmd<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: failed (Result: exit-code) since Thu 2022-12-08 19:51:58 UTC; 2min 33s ago<br> Process: 1299 ExecStart=/usr/sbin/slurmd -D -s $SLURMD_OPTIONS (code=exited, status=1/FAILURE)<br> Main PID: 1299 (code=exited, status=1/FAILURE)<br><br>Dec 08 19:51:49 n0001 systemd[1]: Started Slurm node daemon.<br>Dec 08 19:51:58 n0001 systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL><br>Dec 08 19:51:58 n0001 systemd[1]: slurmd.service: Failed with result 'exit-code'.<br></div><div><br></div><div><br></div><div>Here is the output from grepping through journalctl:</div><div><br></div><div><br></div><div> UNIT=slurmd.service<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> MESSAGE=error: slurmd initialization failed<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Main process exited, code=exited, status=1/FAILURE<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Failed with result 'exit-code'.<br> MESSAGE=Operator of unix-process:911:7771 successfully authenticated as unix-user:root to gain<br> ONE-SHOT authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1.24<br> [systemctl start slurmd] (owned by unix-user:laytonjb)<br> UNIT=slurmd.service<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> MESSAGE=error: slurmd initialization failed<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Main process exited, code=exited, status=1/FAILURE<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Failed with result 'exit-code'.<br> UNIT=slurmd.service<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> MESSAGE=error: slurmd initialization failed<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Main process exited, code=exited, status=1/FAILURE<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Failed with result 'exit-code'.<br> MESSAGE=Operator of unix-process:1254:240421 successfully authenticated as unix-user:root to g<br>ain ONE-SHOT authorization for action org.freedesktop.systemd1.manage-units for system-bus-name::1<br>.47 [systemctl start slurmd] (owned by unix-user:laytonjb)<br> UNIT=slurmd.service<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> MESSAGE=error: slurmd initialization failed<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Main process exited, code=exited, status=1/FAILURE<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Failed with result 'exit-code'.<br> UNIT=slurmd.service<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br> SYSLOG_IDENTIFIER=slurmd<br> _COMM=slurmd<br></div><div> MESSAGE=error: slurmd initialization failed<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Main process exited, code=exited, status=1/FAILURE<br> UNIT=slurmd.service<br> MESSAGE=slurmd.service: Failed with result 'exit-code'.</div><div><br></div><div><br></div><div>These don't look too useful even with debug5 on.</div><div><br></div><div>Any thoughts?</div><div><br></div><div>Thanks!</div><div><br></div><div>Jeff</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 8, 2022 at 2:01 PM Glen MacLachlan <<a href="mailto:maclach@gwu.edu">maclach@gwu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>What does running this on the compute node show? (looks at journal log for past 12 hours) </div><div>journalctl -S -12h -o verbose | grep slurm<br></div><div><br></div><div><div><br></div><div>You may want to increase your debug verbosity to debug5 <a href="https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdDebug" target="_blank">https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdDebug</a> while tracking down this issue.</div><div><div>For reference, see <a href="https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdDebug" target="_blank">https://slurm.schedmd.com/slurm.conf.html#OPT_SlurmdDebug</a></div><div><br></div></div></div><div>You should also address this error to fix logging:</div><div>[2022-12-08T13:12:17.343] error: chdir(/var/log): Permission denied</div><div><br></div><div>by making a directory /var/log/slurm and making the slurm user the owner on both the controller and compute node. Then update your slurm.conf file like this:</div><div># LOGGING<br>SlurmctldLogFile=/var/log/slurm/slurmctld.log<br>SlurmdLogFile=/var/log/slurm/slurmd.log<br></div><div><br></div><div>and then running 'scontrol reconfigure'</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><span style="color:rgb(0,0,0)">Kind Regards,</span></div><div><span style="color:rgb(0,0,0)">Glen</span><br></div><div><span style="color:rgb(7,55,99);font-size:small"><br></span></div><div><span style="color:rgb(7,55,99);font-size:small">==========================================</span></div><div><div style="font-size:13px"><font color="#073763">Glen MacLachlan, PhD</font></div></div><div><i style="color:rgb(7,55,99);font-size:13px">Lead High Performance Computing Engineer </i><br></div></div></div></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div><div><div><div><div><div><div><div><div><div style="font-size:13px"><font color="#073763">Research Technology Services</font></div></div></div></div></div></div></div></div></div></div><div style="font-size:13px"><font color="#073763">The George Washington University<br></font></div><div style="font-size:13px"><font color="#073763">44983 Knoll Square</font></div><div style="font-size:13px"><font color="#073763">Enterprise Hall, 328L</font></div><div style="font-size:13px"><font color="#073763">Ashburn, VA 20147</font></div></blockquote><span style="color:rgb(7,55,99);font-size:small">==========================================</span><br></div><div dir="ltr"><blockquote style="margin:0px 0px 0px 40px;border:medium none;padding:0px"><div><div style="font-size:13px"><div><br></div></div></div></blockquote><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="color:rgb(102,102,102)"><span style="border-collapse:collapse"><div style="font-size:13px;font-family:arial,sans-serif"><div><br><br><br></div></div></span></span></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Dec 8, 2022 at 1:41 PM Jeffrey Layton <<a href="mailto:laytonjb@gmail.com" target="_blank">laytonjb@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Good afternoon,</div><div><br></div><div>I have a very simple two node cluster using Warewulf 4.3. I was following some instructions on how to install the OpenHPC Slurm binaries (server and client). I booted the compute node and the Slurm Server says it's in an unknown state. This hasn't happened to me before but I would like to debug the problem.</div><div><br></div><div>I checked the services on the S:urm server (head node)</div><div><br></div><div>$ systemctl status munge<br>● munge.service - MUNGE authentication service<br> Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)<br> Active: active (running) since Thu 2022-12-08 13:12:10 EST; 4min 42s ago<br> Docs: man:munged(8)<br> Process: 1140 ExecStart=/usr/sbin/munged (code=exited, status=0/SUCCESS)<br> Main PID: 1182 (munged)<br> Tasks: 4 (limit: 48440)<br> Memory: 1.2M<br> CGroup: /system.slice/munge.service<br> └─1182 /usr/sbin/munged<br><br>Dec 08 13:12:10 localhost.localdomain systemd[1]: Starting MUNGE authentication service...<br>Dec 08 13:12:10 localhost.localdomain systemd[1]: Started MUNGE authentication service.<br><br>$ systemctl status slurmctld<br>● slurmctld.service - Slurm controller daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)<br> Active: active (running) since Thu 2022-12-08 13:12:17 EST; 4min 56s ago<br> Main PID: 1518 (slurmctld)<br> Tasks: 10<br> Memory: 23.0M<br> CGroup: /system.slice/slurmctld.service<br> ├─1518 /usr/sbin/slurmctld -D -s<br> └─1555 slurmctld: slurmscriptd<br><br>Dec 08 13:12:17 localhost.localdomain systemd[1]: Started Slurm controller daemon.<br>Dec 08 13:12:17 localhost.localdomain slurmctld[1518]: slurmctld: No parameter for mcs plugin, de><br>Dec 08 13:12:17 localhost.localdomain slurmctld[1518]: slurmctld: mcs: MCSParameters = (null). on><br>Dec 08 13:13:17 localhost.localdomain slurmctld[1518]: slurmctld: SchedulerParameters=default_que></div><div><br></div><div><br></div><div><br></div><div>I then booted the compute node and checked the services there:</div><div><br></div><div>systemctl status munge<br>● munge.service - MUNGE authentication service<br> Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)<br> Active: active (running) since Thu 2022-12-08 18:14:53 UTC; 3min 24s ago<br> Docs: man:munged(8)<br> Process: 786 ExecStart=/usr/sbin/munged (code=exited, status=0/SUCCESS)<br> Main PID: 804 (munged)<br> Tasks: 4 (limit: 26213)<br> Memory: 940.0K<br> CGroup: /system.slice/munge.service<br> └─804 /usr/sbin/munged<br><br>Dec 08 18:14:53 n0001 systemd[1]: Starting MUNGE authentication service...<br>Dec 08 18:14:53 n0001 systemd[1]: Started MUNGE authentication service.<br><br>systemctl status slurmd<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: failed (Result: exit-code) since Thu 2022-12-08 18:15:53 UTC; 2min 40s ago<br> Process: 897 ExecStart=/usr/sbin/slurmd -D -s $SLURMD_OPTIONS (code=exited, status=1/FAILURE)<br> Main PID: 897 (code=exited, status=1/FAILURE)<br><br>Dec 08 18:15:44 n0001 systemd[1]: Started Slurm node daemon.<br>Dec 08 18:15:53 n0001 systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL><br>Dec 08 18:15:53 n0001 systemd[1]: slurmd.service: Failed with result 'exit-code'.<br><br># systemctl status slurmd<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Thu 2022-12-08 18:19:04 UTC; 5s ago<br> Main PID: 996 (slurmd)<br> Tasks: 2<br> Memory: 1012.0K<br> CGroup: /system.slice/slurmd.service<br> ├─996 /usr/sbin/slurmd -D -s --conf-server localhost<br> └─997 /usr/sbin/slurmd -D -s --conf-server localhost<br><br>Dec 08 18:19:04 n0001 systemd[1]: Started Slurm node daemon.</div><div><br></div><div><br></div><div><br></div><div><br></div><div>On the SLurm server I checked the queue and "sinfo -a" and found the following:</div><div><br></div><div><span style="font-family:monospace">$ squeue<br> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br>$ sinfo -a<br>PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>normal* up 1-00:00:00 1 unk* n0001</span><br></div><div><br></div><div><br></div><div>After a few moments (less than a minute - maybe 20-30 seconds, slurmd on the compute node fails. WHen I checked the service I saw this:</div><div><br></div><div><span style="font-family:monospace">$ systemctl status slurmd<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: failed (Result: exit-code) since Thu 2022-12-08 18:19:13 UTC; 10min ago<br> Process: 996 ExecStart=/usr/sbin/slurmd -D -s $SLURMD_OPTIONS (code=exited, status=1/FAILURE)<br> Main PID: 996 (code=exited, status=1/FAILURE)<br><br>Dec 08 18:19:04 n0001 systemd[1]: Started Slurm node daemon.<br>Dec 08 18:19:13 n0001 systemd[1]: slurmd.service: Main process exited, code=exited, status=1/FAIL><br>Dec 08 18:19:13 n0001 systemd[1]: slurmd.service: Failed with result 'exit-code'.</span></div><div><br></div><div><br></div><div>Below are the logs for the slurm server for today (I rebooted the compute twice)</div><div><br></div><div>[2022-12-08T13:12:17.343] error: chdir(/var/log): Permission denied<br>[2022-12-08T13:12:17.343] error: Configured MailProg is invalid<br>[2022-12-08T13:12:17.347] slurmctld version 22.05.2 started on cluster cluster<br>[2022-12-08T13:12:17.371] No memory enforcing mechanism configured.<br>[2022-12-08T13:12:17.374] Recovered state of 1 nodes<br>[2022-12-08T13:12:17.374] Recovered JobId=3 Assoc=0<br>[2022-12-08T13:12:17.374] Recovered JobId=4 Assoc=0<br>[2022-12-08T13:12:17.374] Recovered information about 2 jobs<br>[2022-12-08T13:12:17.375] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions<br>[2022-12-08T13:12:17.375] Recovered state of 0 reservations<br>[2022-12-08T13:12:17.375] read_slurm_conf: backup_controller not specified<br>[2022-12-08T13:12:17.376] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure<br>[2022-12-08T13:12:17.376] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions<br>[2022-12-08T13:12:17.376] Running as primary controller<br>[2022-12-08T13:12:17.376] No parameter for mcs plugin, default values set<br>[2022-12-08T13:12:17.376] mcs: MCSParameters = (null). ondemand set.<br>[2022-12-08T13:13:17.471] SchedulerParameters=default_queue_depth=100,max_rpc_cnt=0,max_sched_time=2,partition_job_depth=0,sched_max_job_start=0,sched_min_interval=2<br>[2022-12-08T13:17:17.940] error: Nodes n0001 not responding<br>[2022-12-08T13:22:17.533] error: Nodes n0001 not responding<br>[2022-12-08T13:27:17.048] error: Nodes n0001 not responding</div><div><br></div><div>There are no logs on the compute node.</div><div><br></div><div>Any suggestions where to start looking? I think I'm seeing the trees and not the forest :)<br></div><div><br></div><div>Thanks!</div><div><br></div><div>Jeff</div><div><br></div><div>P.S Here's some relevant features from the server slurm.conf</div><div><br></div><div><br></div><div># slurm.conf file generated by configurator.html.<br># Put this file on all nodes of your cluster.<br># See the slurm.conf man page for more information.<br>#<br>ClusterName=cluster<br>SlurmctldHost=localhost<br>#SlurmctldHost=</div><div>...</div><div># slurm.conf file generated by configurator.html.<br># Put this file on all nodes of your cluster.<br># See the slurm.conf man page for more information.<br>#<br>ClusterName=cluster<br>SlurmctldHost=localhost<br>#SlurmctldHost=<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Here's some relevant parts of slurm.conf on the client node:</div><div><br></div><div><br></div><div><br></div><div><br></div><div># slurm.conf file generated by configurator.html.<br># Put this file on all nodes of your cluster.<br># See the slurm.conf man page for more information.<br>#<br>ClusterName=cluster<br>SlurmctldHost=localhost<br>#SlurmctldHost=</div><div>...</div><div># slurm.conf file generated by configurator.html.<br># Put this file on all nodes of your cluster.<br># See the slurm.conf man page for more information.<br>#<br>ClusterName=cluster<br>SlurmctldHost=localhost<br>#SlurmctldHost=</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
</blockquote></div>
</blockquote></div></div>