<html><head></head><body><div class="ydp844fa094yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px;"><div><span>Hello, <br></span></div><div><span><br></span></div><div><span>I am trying to install slurm in a small test cluster. Just after installation the nodes were up and running, but after rebooting the machines, the following error appears : <br></span></div><div><span><br></span></div><div><span><span><span> slurmd: debug: switch NONE plugin loaded<br> slurmd: error: Error binding slurm stream socket: Address already in use<br> slurmd: error: Unable to bind listen port (192.168.70.213:8018): Address already in use</span></span><br></span></div><div><span><br></span></div><div><span>Below the details of my history installation and configuration files. If you have the time to take a look and give me an idea on how to solve the problem let me know. <br></span></div><div><span><br></span></div><div><span>Thank you.</span></div><div><span><br></span></div><div><span>Kind regards,<br></span></div><div><span>Alseny</span></div><div><span><br></span></div><div><span>-------------------------------------------------------------------------------</span></div><div><span><br></span></div><div><span># INSTALLATION HISTORY DETAILS<br></span></div><div><span><br></span></div><div><span># mini cluster test set up: </span></div><div><span></span><br></div><span># master node : hostname -> toklap124, IP 192.168.70.214<br># compute node 1 : hostname -> tokwor112, IP 192.168.70.212<br># compute node 2 : hostname -> toklap120, IP 192.168.70.213<br><br><br># All the following commands have been done as root in each of the 3 machines<br><br>############################################################### AT MASTER NODE <br><br># checking IP and hostname <br>ip route get 8.8.8.8 | awk '{print $NF; exit}'<br> # 192.168.70.214<br># checking that the hostname is correct<br>vi /etc/hostname<br> # toklap124<br>vi /etc/hosts<br> # 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4<br> # ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6<br> # 192.168.70.214 toklap124<br><br>############################################################### AT COMPUTE NODE1<br><br># checking IP and hostname <br>ip route get 8.8.8.8 | awk '{print $NF; exit}'<br> # 192.168.70.212<br>vi /etc/hostname<br> # tokwor112 <br>vi /etc/hosts<br> # 127.0.0.1 localhost.localdomain localhost4 localhost4.localdomain4<br> # ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6<br></span><div><span> # 192.168.70.212 tokwor112</span></div><div><span></span><br></div><div><br></div><span>############################################################### AT COMPUTE NODE2<br><br># checking IP and hostname <br>ip route get 8.8.8.8 | awk '{print $NF; exit}'<br> # 192.168.70.213<br>vi /etc/hostname<br> # toklap120<br>vi /etc/hosts<br> # 127.0.0.1 localhost.localdomain localhost4 localhost4.localdomain4<br> # ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6<br> # 192.168.70.213 toklap120<br><br><br>-------------------------------------------------------------- REMOVE PREVIOUS INSTALLATION<br><br>########################################################## ON ALL NODES <br>yum remove mariadb-server mariadb-devel -y<br>yum remove slurm munge munge-libs munge-devel -y<br>userdel -r slurm<br>userdel -r munge<br><br># install maria<br>yum install mariadb-server mariadb-devel -y<br><br>cd /<br># create the new user group "munge" (-g option is used to assign a numerical group ID)<br>export MUNGEUSER=1127<br>groupadd -g $MUNGEUSER munge<br>useradd -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u $MUNGEUSER -g munge -s /sbin/nologin munge<br><br># create the new user group "slurm"<br>export SLURMUSER=1128<br>groupadd -g $SLURMUSER slurm<br>useradd -m -c "SLURM workload manager" -d /var/lib/slurm -u $SLURMUSER -g slurm -s /bin/bash slurm<br><br># check<br>grep '1127' /etc/passwd<br>#OUTPUT -> munge:x:1127:1127:MUNGE Uid 'N' Gid Emporium:/var/lib/munge:/sbin/nologin<br><br>grep '1128' /etc/passwd<br>#OUTPUT -> slurm:x:1128:1128:SLURM workload manager:/var/lib/slurm:/bin/bash<br><br><br>############################################################# BACK TO MASTER NODE<br><br># at master node <br>yum install epel-release -y<br>yum install munge munge-libs munge-devel -y<br>yum install rng-tools -y<br>/usr/sbin/create-munge-key -r # overwrite key? yes<br>dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key<br><br><br>############################################################# AT COMPUTE NODES <br><br># at compute nodes <br>yum install epel-release -y<br>yum install munge munge-libs munge-devel -y<br>rm -rf /etc/munge/munge.key<br><br>############################################################# AT MASTER NODE <br># at master node <br># sending the key to each of the compute nodes <br># compute node 1 tokwor112: <br>scp /etc/munge/munge.key root@192.168.70.212:/etc/munge<br># compute node 2 toklap120:<br>scp /etc/munge/munge.key root@192.168.70.213:/etc/munge<br><br># starting munge services <br>chown munge: /etc/munge/munge.key<br>chmod 400 /etc/munge/munge.key<br>chown -R munge: /etc/munge/ /var/log/munge/<br>chmod 0700 /etc/munge/ /var/log/munge/<br>systemctl enable munge<br>systemctl start munge<br><br>############################################################## AT COMPUTE NODES <br><br>chown munge: /etc/munge/munge.key<br>chmod 400 /etc/munge/munge.key<br>chown -R munge: /etc/munge/ /var/log/munge/<br>chmod 0700 /etc/munge/ /var/log/munge/<br>systemctl enable munge<br>systemctl start munge<br><br>############################################################## AT MASTER <br># testing munge <br>[root@toklap124 /]# munge -n | unmunge<br>STATUS: Success (0)<br>ENCODE_HOST: toklap124 (192.168.70.214)<br>ENCODE_TIME: 2018-12-27 14:44:19 +0900 (1545889459)<br>DECODE_TIME: 2018-12-27 14:44:19 +0900 (1545889459)<br><br><br><br>[root@toklap124 /]# munge -n | ssh 192.168.70.212 unmunge<br>root@192.168.70.212's password: <br>STATUS: Success (0)<br>ENCODE_HOST: ??? (192.168.70.214)<br>ENCODE_TIME: 2018-12-27 14:44:52 +0900 (1545889492)<br>DECODE_TIME: 2018-12-27 14:44:57 +0900 (1545889497)<br><br><br>[root@toklap124 /]# munge -n | ssh 192.168.70.213 unmunge<br>root@192.168.70.213's password: <br>STATUS: Success (0)<br>ENCODE_HOST: ??? (192.168.70.214)<br>ENCODE_TIME: 2018-12-27 14:46:08 +0900 (1545889568)<br>DECODE_TIME: 2018-12-27 14:46:13 +0900 (1545889573)<br><br># OK it is working but are you sure those ENCODE_HOST ??? <br># "???" does not look quite right, anyway let's keep going.<br><br># at compute node 192.168.70.212 : <br>[root@tokwor112 /]# munge -n | unmunge<br>STATUS: Success (0)<br>ENCODE_HOST: tokwor112 (192.168.70.212)<br>ENCODE_TIME: 2018-12-27 14:48:40 +0900 (1545889720)<br>DECODE_TIME: 2018-12-27 14:48:40 +0900 (1545889720)<br><br>[root@toklap120 /]# munge -n | unmunge<br>STATUS: Success (0)<br>ENCODE_HOST: toklap120 (192.168.70.213)<br>ENCODE_TIME: 2018-12-27 14:49:39 +0900 (1545889779)<br>DECODE_TIME: 2018-12-27 14:49:39 +0900 (1545889779)<br><br><br>###################################################### SLURM INSTALLATION <br><br>###################################################### IN EACH NODE (both master and compute)<br><br>yum install gcc gcc-c++ gcc-gfortran kernel-devel -y <br>yum install openssl openssl-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel man2html libibmad libibumad cpanm* -y <br>yum install wget gcc gcc-c++ hdf5 hdf5-devel -y <br>yum install libcurl-devel json-c-devel lz4-devel libibmad-devel libssh2-devel glibc-devel glib2-devel gtk2-devel -y <br>yum install rpmdevtools -y <br>cd ~ <br> <br>######################## at MASTER NODE: <br>rm -rf rpmbuild<br>rpmbuild -ta slurm-17.11.5.tar.bz2<br>libtool --finish /lib64/security<br>rm -rf ~/slurm_rpms/<br>mkdir ~/slurm_rpms<br>mv rpmbuild/RPMS/x86_64/slurm*.rpm ~/slurm_rpms<br><br># sending the rpm to the compute nodes: <br>scp -r ~/slurm_rpms root@192.168.70.212:~/<br>scp -r ~/slurm_rpms root@192.168.70.213:~/<br><br><br>########################################################## IN EACH NODE <br>yum install ntp -y<br>yum install mailx -y <br>yum install ~/slurm_rpms/*.rpm -y <br>vi /etc/slurm/slurm.conf <br># and inside we copy the following : <br><br>############################################################################ slurm.conf start<br># slurm.conf file generated by configurator easy.html. <br># Put this file on all nodes of your cluster. <br># See the slurm.conf man page for more information. <br># <br>ControlMachine=toklap124 <br>ControlAddr=192.168.70.214 <br>#<br># additional suggestions from https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#reboot-option<br>RebootProgram="/usr/sbin/reboot"<br>UnkillableStepTimeout=120<br># end additional suggestions<br># <br>MailProg=/bin/mail <br>MpiDefault=none <br>#MpiParams=ports=#-# <br>ProctrackType=proctrack/cgroup <br>ReturnToService=1 <br>SlurmctldPidFile=/var/run/slurm/slurmctld.pid <br>SlurmctldPort=8017 <br>SlurmdPidFile=/var/run/slurm/slurmd.pid <br>SlurmdPort=8018 <br>SlurmdSpoolDir=/var/spool/slurm <br>SlurmUser=slurm <br>#SlurmdUser=root <br>StateSaveLocation=/var/spool/slurm <br>SwitchType=switch/none <br>TaskPlugin=task/affinity <br>#<br># <br># TIMERS <br>#KillWait=30 <br>#MinJobAge=300 <br>#SlurmctldTimeout=120 <br>#SlurmdTimeout=300 <br># <br># <br># SCHEDULING <br>FastSchedule=1 <br>SchedulerType=sched/backfill <br>SelectType=select/cons_res <br>SelectTypeParameters=CR_Core <br># <br># <br># LOGGING AND ACCOUNTING <br>AccountingStorageType=accounting_storage/none <br>ClusterName=cluster <br>#JobAcctGatherFrequency=30 <br>JobAcctGatherType=jobacct_gather/none <br>#SlurmctldDebug=3 <br>SlurmctldLogFile=/var/log/slurmctld.log <br>#SlurmdDebug=3 <br>SlurmdLogFile=/var/log/slurmd.log <br># <br># <br># COMPUTE NODES <br>NodeName=tokwor112 NodeAddr=192.168.70.212 CPUs=8 Sockets=1 CoresPerSocket=4 ThreadsPerCore=2 State=UNKNOWN<br>NodeName=toklap120 NodeAddr=192.168.70.213 CPUs=4 Sockets=1 CoresPerSocket=2 ThreadsPerCore=2 State=UNKNOWN<br>PartitionName=production Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br>#################################################################### /etc/slurm/slurm.conf ENDS <br><br># we also set the cgroup.conf as follows in each of the nodes <br><br>############################################################## /etc/slurm/cgroup.conf STARTS<br>CgroupAutomount=yes<br>CgroupReleaseAgentDir="/etc/slurm/cgroup"<br><br>ConstrainCores=no<br>ConstrainRAMSpace=yes<br>TaskAffinity=no<br>ConstrainSwapSpace=yes<br>AllowedSwapSpace=0<br>############################################################## /etc/slurm/cgroup.conf ENDS<br><br>########################################### both on cluster and on compute nodes <br>mkdir /var/run/slurm<br>chown slurm: /var/run/slurm<br>chmod 755 /var/run/slurm<br>mkdir /var/spool/slurm<br>chown slurm: /var/spool/slurm<br>chmod 755 /var/spool/slurm<br>slurmd -C<br> # OUTPUT @ master node NodeName=toklap124 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=2 ThreadsPerCore=2 RealMemory=7727 <br> # OUTPUT @ compute node1 NodeName=tokwor112 CPUs=8 Boards=1 SocketsPerBoard=1 CoresPerSocket=4 ThreadsPerCore=2 RealMemory=15811<br> # OUTPUT @ compute node2 NodeName=toklap120 CPUs=4 Boards=1 SocketsPerBoard=1 CoresPerSocket=2 ThreadsPerCore=2 RealMemory=7728<br><br>sed -i -e 's/PIDFile=.*/PIDFile=\/var\/run\/slurm\/slurmctld.pid/g' /usr/lib/systemd/system/slurmctld.service<br>sed -i -e 's/PIDFile=.*/PIDFile=\/var\/run\/slurm\/slurmd.pid/g' /usr/lib/systemd/system/slurmd.service<br><br><br># start services ############################## master node only <br>systemctl enable slurmctld<br>systemctl start slurmctld<br>systemctl status slurmctld.service<br># OK now master node is up and running<br> # OUTPUT<br> #● slurmctld.service - Slurm controller daemon<br> # Loaded: loaded (/usr/lib/systemd/system/slurmctld.service; enabled; vendor preset: disabled)<br> # Active: active (running) since Tue 2019-01-08 14:04:45 JST; 491ms ago<br> # Process: 30750 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)<br> # Main PID: 30753 (slurmctld)<br> # Tasks: 7<br> # CGroup: /system.slice/slurmctld.service<br> # └─30753 /usr/sbin/slurmctld<br> # Jan 08 14:04:45 toklap124 systemd[1]: Starting Slurm controller daemon...<br> # Jan 08 14:04:45 toklap124 systemd[1]: Started Slurm controller daemon.<br><br><br># start services ############################## compute nodes only<br>systemctl enable slurmd.service<br>systemctl start slurmd.service<br>systemctl status slurmd.service<br><br> # OUTPUT <br><br> # COMPUTE NODE 1<br> # ● slurmd.service - Slurm node daemon<br> # Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> # Active: active (running) since Tue 2019-01-08 14:05:09 JST; 453ms ago<br> # Process: 22335 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br> # Main PID: 22340 (slurmd)<br> # Tasks: 2<br> # CGroup: /system.slice/slurmd.service<br> # ├─ 4960 /usr/sbin/slurmd<br> # └─22340 /usr/sbin/slurmd<br> #<br> # Jan 08 14:05:09 tokwor112 systemd[1]: Starting Slurm node daemon...<br> # Jan 08 14:05:09 tokwor112 systemd[1]: PID file /var/run/slurm/slurmd.pid not readable (yet?) after start.<br> # Jan 08 14:05:09 tokwor112 systemd[1]: Started Slurm node daemon.<br><br> # COMPUTE NODE 2<br> # ● slurmd.service - Slurm node daemon<br> # Loaded: loaded (/usr/lib/systemd/system/slurmd.service; enabled; vendor preset: disabled)<br> # Active: active (running) since Tue 2019-01-08 14:05:17 JST; 541ms ago<br> # Process: 7873 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br> # Main PID: 7878 (slurmd)<br> # Tasks: 3<br> # CGroup: /system.slice/slurmd.service<br> # ├─1236 /usr/sbin/slurmd<br> # └─7878 /usr/sbin/slurmd<br> #<br> # Jan 08 14:05:17 toklap120 systemd[1]: Starting Slurm node daemon...<br> # Jan 08 14:05:17 toklap120 systemd[1]: Started Slurm node daemon.<br><br><br>############################################# AT NODE 1<br># now compute node 1 is up and running<br># you can launch sinfo successfully <br>sinfo<br> OUTPUT <br> # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br> # production* up infinite 2 idle toklap120,tokwor112<br><br><br>############################################# AT NODE 2<br>sinfo<br> OUTPUT <br> # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br> # production* up infinite 2 idle toklap120,tokwor112<br><br><br>################################################################### COMPUTE NODE 2 job test <br>[root@toklap120 ~]# srun --ntasks=1 --label /bin/hostname && pwd && whoami<br># OUTPUT <br> 0: toklap120<br> /root/testSlurm<br> root<br><br><br> cd ~<br> mkdir testSlurm<br> cd testSlurm/<br> vi job.slurm<br>and inside we copy : <br><br>#!/bin/bash<br>#SBATCH -J pbe_delta # Job name<br>#SBATCH -o pbe_delta.o%j # Name of stdout output file(%j expands to jobId)<br>#SBATCH -e pbe_delta.o%j # Name of stderr output file(%j expands to jobId)<br>#SBATCH -N 1 # Total number of nodes requested (16 cores/node)<br>#SBATCH -n 1<br>#SBATCH -t 48:00:00 # Run time (hh:mm:ss) <br>date> output.out<br>pwd >> output.out<br>hostname >> output.out<br>ls -lah<br><br><br># launching the job <br>sbatch job.slurm<br><br># getting the message <br># Submitted batch job 17<br><br># inside the directory 2 new files are present: <br> output.out pbe_delta.o17<br># Output.out contains the date, pwd e hostname like in the submitted job:<br> Tue 8 Jan 14:22:57 JST 2019<br> /root/testSlurm<br> toklap120<br><br><br>############################## >> REBOOT MACHINE TESTING <br><br># Rebooting each node <br>reboot now <br><br># AFTER RESTART COMPLETES <br>################################################## IN EACH COMPUTE NODE<br># disabling firewall on the compute nodes<br>systemctl stop firewalld<br>systemctl disable firewalld # Ok no errors<br># syncronizing clocks<br>chkconfig ntpd on<br>ntpdate pool.ntp.org<br>systemctl start ntpd # clock should be now syncronized<br>systemctl enable munge<br>systemctl start munge<br>systemctl enable slurmd<br>systemctl stop slurmd<br>systemctl start slurmd<br>systemctl status slurmd<br>slurmd -D -vvv<br> # OUTPUT <br> # from NODE 1 <br> slurmd: debug: switch NONE plugin loaded<br> slurmd: error: Error binding slurm stream socket: Address already in use<br> slurmd: error: Unable to bind listen port (192.168.70.212:8018): Address already in use<br><br> # from NODE 2<br> slurmd: debug: switch NONE plugin loaded<br> slurmd: error: Error binding slurm stream socket: Address already in use<br> slurmd: error: Unable to bind listen port (192.168.70.213:8018): Address already in use<br><br><br># not working anymore <br><br></span></div></body></html>