[slurm-users] failed to send msg type 6002: No route to host

[Patrick Bégou]

Hi,

I'm new to slurm (as admin) and I need some help. Testing my initial
setup with:

    [begou at tenibre ~]$ *salloc -n 1 sh*
    salloc: Granted job allocation 11
    sh-4.4$ *squeue*
                 JOBID PARTITION     NAME     USER ST       TIME  NODES
    NODELIST(REASON)
                    *11 *      all       sh    begou  R       0:16     
    1 tenibre-0-0
    sh-4.4$*srun /usr/bin/hostname*
    srun: error: timeout waiting for task launch, started 0 of 1 tasks
    srun: Job step 11.0 aborted before step completely launched.
    srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
    srun: error: Timed out waiting for job step to complete

I check the connections:

*tenibre is the login node* (no daemon running)

    nc -v tenibre-0-0 6818
    nc -v management1 6817

*management1 is the management node* (slurmctld running)

    nc -v tenibre-0-0 6818

*tenibre-0-0 is the first compute node* (slurmd running)

    nc -v management1 6817

All tests return "/Ncat: Connected.../"

The command "id begou" works on all nodes and I can reach my home
directory on the login node and on the compute node.

On the compute node slurmd.log shows:

    [2020-11-10T11:21:38.050]*launch task* *11.0 *request from UID:23455
    GID:1036 HOST:172.30.1.254 PORT:42220
    [2020-11-10T11:21:38.050] debug:  Checking credential with 508 bytes
    of sig data
    [2020-11-10T11:21:38.050] _run_prolog: run job script took usec=12
    [2020-11-10T11:21:38.050] _run_prolog: prolog with lock for job 11
    ran for 0 seconds
    [2020-11-10T11:21:38.053] debug:  AcctGatherEnergy NONE plugin loaded
    [2020-11-10T11:21:38.053] debug:  AcctGatherProfile NONE plugin loaded
    [2020-11-10T11:21:38.053] debug:  AcctGatherInterconnect NONE plugin
    loaded
    [2020-11-10T11:21:38.053] debug:  AcctGatherFilesystem NONE plugin
    loaded
    [2020-11-10T11:21:38.053] debug:  switch NONE plugin loaded
    [2020-11-10T11:21:38.054] [11.0] debug:  Job accounting gather
    NOT_INVOKED plugin loaded
    [2020-11-10T11:21:38.054] [11.0] debug:  Message thread started pid
    = 12099
    [2020-11-10T11:21:38.054] debug:  task_p_slurmd_reserve_resources: 11 0
    [2020-11-10T11:21:38.068] [11.0] debug:  task NONE plugin loaded
    [2020-11-10T11:21:38.068] [11.0] debug:  Checkpoint plugin loaded:
    checkpoint/none
    [2020-11-10T11:21:38.068] [11.0] Munge credential signature plugin
    loaded
    [2020-11-10T11:21:38.068] [11.0] debug:  job_container none plugin
    loaded
    [2020-11-10T11:21:38.068] [11.0] debug:  mpi type = pmi2
    [2020-11-10T11:21:38.068] [11.0] debug:  xcgroup_instantiate: cgroup
    '/sys/fs/cgroup/freezer/slurm' already exists
    [2020-11-10T11:21:38.068] [11.0] debug:  spank: opening plugin stack
    /etc/slurm/plugstack.conf
    [2020-11-10T11:21:38.068] [11.0] debug:  mpi type = (null)
    [2020-11-10T11:21:38.068] [11.0] debug:  using mpi/pmi2
    [2020-11-10T11:21:38.068] [11.0] debug:  _setup_stepd_job_info:
    SLURM_STEP_RESV_PORTS not found in env
    [2020-11-10T11:21:38.068] [11.0] debug:  mpi/pmi2: setup sockets
    [2020-11-10T11:21:38.069] [11.0] debug:  mpi/pmi2: started agent thread
    [2020-11-10T11:21:38.069] [11.0]*error: connect io: No route to host*
    [2020-11-10T11:21:38.069] [11.0] error: IO setup failed: No route to
    host
    [2020-11-10T11:21:38.069] [11.0] debug:  step_terminate_monitor_stop
    signaling condition
    [2020-11-10T11:21:38.069] [11.0] error: job_manager exiting
    abnormally, rc = 4021
    [2020-11-10T11:21:38.069] [11.0] debug:  Sending launch resp rc=4021
    [2020-11-10T11:21:38.069] [11.0] debug:  _send_srun_resp_msg: 0/5
    *failed to send msg type 6002: No route to host*
    [2020-11-10T11:21:38.169] [11.0] debug:  _send_srun_resp_msg: 1/5
    failed to send msg type 6002: No route to host
    [2020-11-10T11:21:38.370] [11.0] debug:  _send_srun_resp_msg: 2/5
    failed to send msg type 6002: No route to host
    [2020-11-10T11:21:38.770] [11.0] debug:  _send_srun_resp_msg: 3/5
    failed to send msg type 6002: No route to host
    [2020-11-10T11:21:39.570] [11.0] debug:  _send_srun_resp_msg: 4/5
    failed to send msg type 6002: No route to host
    [2020-11-10T11:21:40.370] [11.0] debug:  _send_srun_resp_msg: 5/5
    failed to send msg type 6002: No route to host
    [2020-11-10T11:21:40.372] [11.0] debug:  Message thread exited
    [2020-11-10T11:21:40.372] [11.0] debug:  mpi/pmi2: agent thread exit
    [2020-11-10T11:21:40.372] [11.0] *done with job*


But I do not understand what this "No route to host" means.


Thanks for your help.

Patrick


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201110/30e6cca9/attachment.htm>