[slurm-users] failed to send msg type 6002: No route to host

[Pocina, Goran]

I think this message can also happen if the slurm.conf on your login node is missing the entry for the slurmd node.  2020 versions have a way to automate sync of the configuration.

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Patrick Bégou
Sent: Thursday, November 12, 2020 7:38 AM
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] failed to send msg type 6002: No route to host


This message was sent by an external party.

Hi slurm admins and developpers,

no one has an idea about this problem ?

Still investigating this morning I discover that it works from the management node (a small VM running slurmctld) even if I have no home directory on it (I use a su command from root to gain unprivileged user setup). It still doesn't run from the login node even with all firewall disabled :-(

Patrick

Le 10/11/2020 à 11:54, Patrick Bégou a écrit :

Hi,

I'm new to slurm (as admin) and I need some help. Testing my initial setup with:
[begou at tenibre ~]$ salloc -n 1 sh
salloc: Granted job allocation 11
sh-4.4$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                11       all       sh    begou  R       0:16      1 tenibre-0-0
sh-4.4$ srun /usr/bin/hostname
srun: error: timeout waiting for task launch, started 0 of 1 tasks
srun: Job step 11.0 aborted before step completely launched.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: Timed out waiting for job step to complete

I check the connections:

tenibre is the login node (no daemon running)
nc -v tenibre-0-0 6818
nc -v management1 6817
management1 is the management node (slurmctld running)
nc -v tenibre-0-0 6818
tenibre-0-0 is the first compute node (slurmd running)

nc -v management1 6817

All tests return "Ncat: Connected..."

The command "id begou" works on all nodes and I can reach my home directory on the login node and on the compute node.

On the compute node slurmd.log shows:
[2020-11-10T11:21:38.050] launch task 11.0 request from UID:23455 GID:1036 HOST:172.30.1.254 PORT:42220
[2020-11-10T11:21:38.050] debug:  Checking credential with 508 bytes of sig data
[2020-11-10T11:21:38.050] _run_prolog: run job script took usec=12
[2020-11-10T11:21:38.050] _run_prolog: prolog with lock for job 11 ran for 0 seconds
[2020-11-10T11:21:38.053] debug:  AcctGatherEnergy NONE plugin loaded
[2020-11-10T11:21:38.053] debug:  AcctGatherProfile NONE plugin loaded
[2020-11-10T11:21:38.053] debug:  AcctGatherInterconnect NONE plugin loaded
[2020-11-10T11:21:38.053] debug:  AcctGatherFilesystem NONE plugin loaded
[2020-11-10T11:21:38.053] debug:  switch NONE plugin loaded
[2020-11-10T11:21:38.054] [11.0] debug:  Job accounting gather NOT_INVOKED plugin loaded
[2020-11-10T11:21:38.054] [11.0] debug:  Message thread started pid = 12099
[2020-11-10T11:21:38.054] debug:  task_p_slurmd_reserve_resources: 11 0
[2020-11-10T11:21:38.068] [11.0] debug:  task NONE plugin loaded
[2020-11-10T11:21:38.068] [11.0] debug:  Checkpoint plugin loaded: checkpoint/none
[2020-11-10T11:21:38.068] [11.0] Munge credential signature plugin loaded
[2020-11-10T11:21:38.068] [11.0] debug:  job_container none plugin loaded
[2020-11-10T11:21:38.068] [11.0] debug:  mpi type = pmi2
[2020-11-10T11:21:38.068] [11.0] debug:  xcgroup_instantiate: cgroup '/sys/fs/cgroup/freezer/slurm' already exists
[2020-11-10T11:21:38.068] [11.0] debug:  spank: opening plugin stack /etc/slurm/plugstack.conf
[2020-11-10T11:21:38.068] [11.0] debug:  mpi type = (null)
[2020-11-10T11:21:38.068] [11.0] debug:  using mpi/pmi2
[2020-11-10T11:21:38.068] [11.0] debug:  _setup_stepd_job_info: SLURM_STEP_RESV_PORTS not found in env
[2020-11-10T11:21:38.068] [11.0] debug:  mpi/pmi2: setup sockets
[2020-11-10T11:21:38.069] [11.0] debug:  mpi/pmi2: started agent thread
[2020-11-10T11:21:38.069] [11.0] error: connect io: No route to host
[2020-11-10T11:21:38.069] [11.0] error: IO setup failed: No route to host
[2020-11-10T11:21:38.069] [11.0] debug:  step_terminate_monitor_stop signaling condition
[2020-11-10T11:21:38.069] [11.0] error: job_manager exiting abnormally, rc = 4021
[2020-11-10T11:21:38.069] [11.0] debug:  Sending launch resp rc=4021
[2020-11-10T11:21:38.069] [11.0] debug:  _send_srun_resp_msg: 0/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:38.169] [11.0] debug:  _send_srun_resp_msg: 1/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:38.370] [11.0] debug:  _send_srun_resp_msg: 2/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:38.770] [11.0] debug:  _send_srun_resp_msg: 3/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:39.570] [11.0] debug:  _send_srun_resp_msg: 4/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:40.370] [11.0] debug:  _send_srun_resp_msg: 5/5 failed to send msg type 6002: No route to host
[2020-11-10T11:21:40.372] [11.0] debug:  Message thread exited
[2020-11-10T11:21:40.372] [11.0] debug:  mpi/pmi2: agent thread exit
[2020-11-10T11:21:40.372] [11.0] done with job



But I do not understand what this "No route to host" means.



Thanks for your help.

Patrick




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201112/f643317d/attachment-0001.htm>