<div dir="ltr"><div dir="ltr">Hi,<br><div><br></div><div>Thanks for your support for configuring Slurm -
<b>Benson Muite, Michael Smith and Marcus Wagner
</b><br></div><div><b><br></b></div><div>Finally I am able to set up Slurm on master and compute nodes with given instructions - ntp, hostname file and firewalls settings I have followed and corrected.</div><div><br></div><div>[root@smaster ~]# sinfo<br>PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>debug up infinite 1 idle snode<br>hpc* up infinite 1 idle smaster<br>[root@smaster ~]#<br></div><div><br></div><div>Regards,</div><div>Zain</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 2, 2021 at 6:35 PM Zainul Abiddin <<a href="mailto:zainul1114@gmail.com">zainul1114@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All,<div>I have done slurmdbd configuration and while i am trying to run account manager with <b>sacct</b> i am getting below error.</div><div><br></div><div>[root@smaster ~]# sacct<br>sacct: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:localhost:6819: Connection refused<br>sacct: error: Sending PersistInit msg: Connection refused<br>sacct: error: Problem talking to the database: Connection refused<br>[root@smaster ~]#<br></div><div><br></div><div>My slurmdbd configuration :</div><div>[root@smaster ~]# cat /etc/slurm/slurmdbd.conf<br>AuthType=auth/munge<br>DbdAddr=localhost<br>DbdHost=localhost<br>SlurmUser=slurm<br>DebugLevel=4<br>LogFile=/var/log/slurm/slurmdbd.log<br>PidFile=/var/run/slurmdbd.pid<br>StorageType=accounting_storage/mysql<br>StorageHost=localhost<br>StoragePass=password<br>StorageUser=slurm<br>StorageLoc=slurm_acct_db<br><br>[root@smaster ~]# chown slurm: /etc/slurm/slurmdbd.conf<br>[root@smaster ~]# chmod 600 /etc/slurm/slurmdbd.conf<br>[root@smaster ~]# mkdir /var/log/slurm<br>[root@smaster ~]# touch /var/log/slurm/slurmdbd.log<br>[root@smaster ~]# chown slurm: /var/log/slurm/slurmdbd.log<br>[root@smaster ~]# scontrol show config | grep AccountingStorageHost<br>AccountingStorageHost = localhost<br><br>Note:<br>i have edited file /etc/slurm/slurm.conf and modified the below line<br># LOGGING AND ACCOUNTING<br>AccountingStorageType=accounting_storage/slurmdbd<br>Then restarted all the services<br><br>[root@smaster ~]# for i in munge slurmd slurmctld slurmdbd; do service $i status; done<br>Redirecting to /bin/systemctl status munge.service<br>● munge.service - MUNGE authentication service<br> Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:10 IST; 3h 36min ago<br> Docs: man:munged(8)<br> Main PID: 20613 (munged)<br> CGroup: /system.slice/munge.service<br> └─20613 /usr/sbin/munged<br><br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Stopped MUNGE authentication service.<br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Starting MUNGE authentication service...<br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started MUNGE authentication service.<br>Redirecting to /bin/systemctl status slurmd.service<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:10 IST; 3h 36min ago<br> Main PID: 20637 (slurmd)<br> CGroup: /system.slice/slurmd.service<br> └─20637 /usr/sbin/slurmd -D<br><br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started Slurm node daemon.<br>Feb 02 15:30:47 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 7 for UID 0<br>Feb 02 15:31:46 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 8 for UID 0<br>Feb 02 15:33:43 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 9 for UID 0</div><div><br>Redirecting to /bin/systemctl status slurmctld.service<br>● slurmctld.service - Slurm controller daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:11 IST; 3h 36min ago<br> Main PID: 20660 (slurmctld)<br> CGroup: /system.slice/slurmctld.service<br> └─20660 /usr/sbin/slurmctld -D<br><br>Feb 02 13:21:11 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started Slurm controller daemon.<br>Redirecting to /bin/systemctl status slurmdbd.service<br>● slurmdbd.service - Slurm DBD accounting daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 16:29:11 IST; 28min ago<br> Main PID: 24146 (slurmdbd)<br> CGroup: /system.slice/slurmdbd.service<br> └─24146 /usr/sbin/slurmdbd -D<br><br>Feb 02 16:29:11 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started Slurm DBD accounting daemon.<br>[root@smaster ~]# srun --ntasks=2 --label /bin/hostname<br>srun: job 22 queued and waiting for resources<br>srun: job 22 has been allocated resources<br>1: <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a><br>0: <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a><br>[root@smaster ~]#<br><br><br>However when i run the below command </div><div><br></div><div>[root@smaster ~]# sacct<br>sacct: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:localhost:6819: Connection refused<br>sacct: error: Sending PersistInit msg: Connection refused<br>sacct: error: Problem talking to the database: Connection refused<br>[root@smaster ~]#<br></div><div><br></div><div>and i have troubleshooted below steps</div><div><br></div>[root@smaster ~]# telnet localhost 6819<br>Trying ::1...<br>telnet: connect to address ::1: Connection refused<br>Trying 127.0.0.1...<br>telnet: connect to address <a href="http://127.0.0.1/" target="_blank">127.0.0.1</a>: Connection refused<br><div>[root@smaster ~]#</div><div><br></div>[root@smaster ~]# mysql -p -u slurm slurm_acct_db<br>Enter password:<br>Welcome to the MariaDB monitor. Commands end with ; or \g.<br>Your MariaDB connection id is 9<br>Server version: 10.1.48-MariaDB MariaDB Server<br><br>Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.<br><br>Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.<br><br>MariaDB [slurm_acct_db]> show tables;<br>Empty set (0.00 sec)<br><br>MariaDB [slurm_acct_db]><br><div><br></div><div>Then i have added DBPort and restarted services </div>[root@smaster ~]# cat /etc/slurm/slurmdbd.conf<br>AuthType=auth/munge<br>DbdAddr=localhost<br>DbdHost=localhost<br><b>DbdPort=6819</b><br>SlurmUser=slurm<br>DebugLevel=4<br>LogFile=/var/log/slurm/slurmdbd.log<br>PidFile=/var/run/slurmdbd.pid<br>StorageType=accounting_storage/mysql<br>StorageHost=localhost<br>StoragePass=password<br>StorageUser=slurm<br>StorageLoc=slurm_acct_db<br><div>[root@smaster ~]#</div><div><br></div><div>[root@smaster ~]# for i in munge slurmd slurmctld slurmdbd; do service $i status; done<br>Redirecting to /bin/systemctl status munge.service<br>● munge.service - MUNGE authentication service<br> Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:10 IST; 3h 55min ago<br> Docs: man:munged(8)<br> Main PID: 20613 (munged)<br> CGroup: /system.slice/munge.service<br> └─20613 /usr/sbin/munged<br><br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Stopped MUNGE authentication service.<br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Starting MUNGE authentication service...<br>Feb 02 13:21:10 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started MUNGE authentication service.<br>Redirecting to /bin/systemctl status slurmd.service<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:10 IST; 3h 55min ago<br> Main PID: 20637 (slurmd)<br> CGroup: /system.slice/slurmd.service<br> └─20637 /usr/sbin/slurmd -D<br><br>Feb 02 15:30:47 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 7 for UID 0<br>Feb 02 15:31:46 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 8 for UID 0<br>Feb 02 15:33:43 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 9 for UID 0<br>Feb 02 15:38:45 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> slurmd[20637]: slurmd: Launching batch job 12 for UID 0<br><br></div><div>Redirecting to /bin/systemctl status slurmctld.service<br>● slurmctld.service - Slurm controller daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 13:21:11 IST; 3h 55min ago<br> Main PID: 20660 (slurmctld)<br> CGroup: /system.slice/slurmctld.service<br> └─20660 /usr/sbin/slurmctld -D<br><br>Feb 02 13:21:11 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started Slurm controller daemon.<br>Redirecting to /bin/systemctl status slurmdbd.service<br>● slurmdbd.service - Slurm DBD accounting daemon<br> Loaded: loaded (/usr/lib/systemd/system/slurmdbd.service; enabled; vendor preset: disabled)<br> Active: active (running) since Tue 2021-02-02 16:29:11 IST; 47min ago<br> Main PID: 24146 (slurmdbd)<br> CGroup: /system.slice/slurmdbd.service<br> └─24146 /usr/sbin/slurmdbd -D<br><br>Feb 02 16:29:11 <a href="http://smaster.calligotech.com/" target="_blank">smaster.calligotech.com</a> systemd[1]: Started Slurm DBD accounting daemon.<br>[root@smaster ~]# ps -ef |grep slurm<br>root 20637 1 0 13:21 ? 00:00:00 /usr/sbin/slurmd -D<br>slurm 20660 1 0 13:21 ? 00:00:08 /usr/sbin/slurmctld -D<br>root 24146 1 0 16:29 ? 00:00:00 /usr/sbin/slurmdbd -D<br>root 25395 18378 0 17:17 pts/2 00:00:00 grep --color=auto slurm<br>[root@smaster ~]# sacct<br>sacct: error: slurm_persist_conn_open_without_init: failed to open persistent connection to host:localhost:6819: Connection refused<br>sacct: error: Sending PersistInit msg: Connection refused<br>sacct: error: Problem talking to the database: Connection refused<br>[root@smaster ~]#<br></div><div><br></div><div>[root@smaster ~]# tail /var/log/slurm/slurmdbd.log<br>[2021-02-02T17:16:01.913] error: mysql_real_connect failed: 2005 Unknown MySQL server host 'smater' (-2)<br>[2021-02-02T17:16:01.913] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.<br>[2021-02-02T17:16:06.963] error: mysql_real_connect failed: 2005 Unknown MySQL server host 'smater' (-2)<br>[2021-02-02T17:16:06.963] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.<br>[2021-02-02T17:16:12.083] error: mysql_real_connect failed: 2005 Unknown MySQL server host 'smater' (-2)<br>[2021-02-02T17:16:12.083] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.<br>[2021-02-02T17:16:17.140] error: mysql_real_connect failed: 2005 Unknown MySQL server host 'smater' (-2)<br>[2021-02-02T17:16:17.141] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.<br>[2021-02-02T17:16:22.804] error: mysql_real_connect failed: 2005 Unknown MySQL server host 'smater' (-2)<br>[2021-02-02T17:16:22.804] error: The database must be up when starting the MYSQL plugin. Trying again in 5 seconds.<br>[root@smaster ~]#</div><div> <br></div><div>Still the problem remains the same. Please help me to resolve this issue.</div><font color="#888888"><div> </div><div>Regards,</div><div>Zain</div></font></div>
</blockquote></div><br clear="all"><div><br></div><br></div>