[slurm-users] srun problem -- Can't find an address, check slurm.conf

[Scott Hazelhurst]

Dear list

We have a relatively new installation of SLURM. We have started to have a problem with some of the nodes when using srun

[scott at cream-ce ~]$ srun --pty -w n38 hostname
srun: error: fwd_tree_thread: can't find address for host n38, check slurm.conf
srun: error: Task launch for 18710.0 failed on node n38: Can't find an address, check slurm.conf
srun: error: Application launch failed: Can't find an address, check slurm.conf
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete


I’ve spent most of the day trying to follow others who’ve had similar problems and check everything but I haven’t made any progress

— Using sbatch there is no problem: jobs launch on the given node and finish normally and reliably

— srun works fine for most of the nodes

— the slum.conf file is identical on all nodes (checked by diffing and no complaints in the logs)

— both the slurmctld and the slurmd start cleanly with no obvious errors or warnings (e.g. about slurm.conf)

— sinfo reports that all our nodes are up, some busy some not. The problem is independent of load on the nodes

— I’ve increased the log level on the control daemon and there’s no obvious additional information when the srun happens

— we use puppet to maintain our infrastructure so while there must be a difference between the  machines that work and those that don’t I can see it.

— all nodes run ntpd and the times appear the same when checked manually

— all nodes have plenty of disk space

— I’ve tried restarting both the slurm and control daemons and this has no effect, even for a short time

— hostname on working and problematic nodes give the expected results in the same format as other

— all hostnames are in /etc/hosts on all machines

— we currently have just less than 40 worker nodes, treewidth=50


We’re running SLURM 17.11.10 under CentOS 7.5


This is the final part of the slurm.conf file

NodeName=n[02,08,10,29-40,42-45] RealMemory=131072 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[05] RealMemory=256000 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[07] RealMemory=45000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[15] RealMemory=48000 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
NodeName=n[16] RealMemory=31000 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[17] RealMemory=215000 Sockets=16 CoresPerSocket=8 ThreadsPerCore=1 State=UNKNOWN
NodeName=n[18] RealMemory=90000 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
NodeName=n[19] RealMemory=515000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[20] RealMemory=64000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[21] RealMemory=64000 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[22] RealMemory=56000 Sockets=16 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
NodeName=n[23] RealMemory=225500 Sockets=20 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN
NodeName=n[27] RealMemory=65536 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN
NodeName=n[28] RealMemory=65536 Sockets=2 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN

PartitionName=batch Nodes=n[02,05,07-08,10,15-23,27-40,42-45] Default=YES MaxTime=40320 State=UP

As examples
— fails: n15, n17,  n27, n28, n38, n45
— success: n02, n10, n16, n18, n29 onwards except for n38, n45


Many thanks for any help

Scott


This communication is intended for the addressee only. It is confidential. If you have received this communication in error, please notify us immediately and destroy the original message. You may not copy or disseminate this communication without the permission of the University. Only authorised signatories are competent to enter into agreements on behalf of the University and recipients are thus advised that the content of this message may not be legally binding on the University and may contain the personal views and opinions of the author, which are not necessarily the views and opinions of The University of the Witwatersrand, Johannesburg. All agreements between the University and outsiders are subject to South African Law unless the University agrees in writing to the contrary.