<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<body>
<div dir="auto">
<div dir="auto">Gentle bump on this, if anyone has suggestions as I weed through the scattered slurm docs. :) </div><div dir="auto"><br></div><div dir="auto">Thanks, </div><div dir="auto">Andrej</div><div dir='auto'><br></div>
<div id="aqm-original" style="color: black;">
<div dir="auto">On February 2, 2021 00:14:37 Andrej Prsa <aprsa09@gmail.com> wrote:</div>
<div><br></div>
<blockquote type="cite" class="gmail_quote" style="margin: 0 0 0 0.75ex; border-left: 1px solid #808080; padding-left: 0.75ex;">
<div dir="auto">Dear list,</div>
<div dir="auto"><br></div>
<div dir="auto">I'm struggling with what seems to be very similar to this thread:</div>
<div dir="auto"><br></div>
<div dir="auto">https://lists.schedmd.com/pipermail/slurm-users/2019-July/003746.html</div>
<div dir="auto"><br></div>
<div dir="auto">I'm using slurm 20.11.3 patched with this fix to detect pmixv4:</div>
<div dir="auto"><br></div>
<div dir="auto"> https://bugs.schedmd.com/show_bug.cgi?id=10683</div>
<div dir="auto"><br></div>
<div dir="auto">and this is what I'm seeing:</div>
<div dir="auto"><br></div>
<div dir="auto">andrej@terra:~$ salloc -N 2 -n 2</div>
<div dir="auto">salloc: Granted job allocation 841</div>
<div dir="auto">andrej@terra:~$ srun hostname</div>
<div dir="auto">srun: launch/slurm: launch_p_step_launch: StepId=841.0 aborted before </div>
<div dir="auto">step completely launched.</div>
<div dir="auto">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</div>
<div dir="auto">srun: error: task 0 launch failed: Unspecified error</div>
<div dir="auto">srun: error: task 1 launch failed: Unspecified error</div>
<div dir="auto"><br></div>
<div dir="auto">In slurmctld.log I have this:</div>
<div dir="auto"><br></div>
<div dir="auto">[2021-02-01T23:58:13.683] sched: _slurm_rpc_allocate_resources JobId=841 </div>
<div dir="auto">NodeList=node[9-10] usec=572</div>
<div dir="auto">[2021-02-01T23:58:19.817] error: mpi_hook_slurmstepd_prefork failure for </div>
<div dir="auto">0x557e7480bcb0s on node9</div>
<div dir="auto">[2021-02-01T23:58:19.829] error: mpi_hook_slurmstepd_prefork failure for </div>
<div dir="auto">0x55f568e00cb0s on node10</div>
<div dir="auto"><br></div>
<div dir="auto">and in slurmd.log I have this for node9:</div>
<div dir="auto"><br></div>
<div dir="auto">[2021-02-01T23:58:19.788] launch task StepId=841.0 request from UID:1000 </div>
<div dir="auto">GID:1000 HOST:192.168.1.1 PORT:35508</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: lllp_distribution: JobId=841 </div>
<div dir="auto">implicit auto binding: cores, dist 1</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: _task_layout_lllp_cyclic: </div>
<div dir="auto">_task_layout_lllp_cyclic</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: _lllp_generate_cpu_bind: </div>
<div dir="auto">_lllp_generate_cpu_bind jobid [841]: mask_cpu, 0x000000000001000000000001</div>
<div dir="auto">[2021-02-01T23:58:19.814] [841.0] error: node9 [0] pmixp_utils.c:108 </div>
<div dir="auto">[pmixp_usock_create_srv] mpi/pmix: ERROR: Cannot bind() UNIX socket </div>
<div dir="auto">/var/spool/slurmd/stepd.slurm.pmix.841.0: Address already in use (98)</div>
<div dir="auto">[2021-02-01T23:58:19.814] [841.0] error: node9 [0] pmixp_server.c:387 </div>
<div dir="auto">[pmixp_stepd_init] mpi/pmix: ERROR: pmixp_usock_create_srv</div>
<div dir="auto">[2021-02-01T23:58:19.814] [841.0] error: (null) [0] mpi_pmix.c:169 </div>
<div dir="auto">[p_mpi_hook_slurmstepd_prefork] mpi/pmix: ERROR: pmixp_stepd_init() failed</div>
<div dir="auto">[2021-02-01T23:58:19.817] [841.0] error: Failed mpi_hook_slurmstepd_prefork</div>
<div dir="auto">[2021-02-01T23:58:19.845] [841.0] error: job_manager exiting abnormally, </div>
<div dir="auto">rc = -1</div>
<div dir="auto">[2021-02-01T23:58:19.892] [841.0] done with job</div>
<div dir="auto"><br></div>
<div dir="auto">and this for node10:</div>
<div dir="auto"><br></div>
<div dir="auto">[2021-02-01T23:58:19.788] launch task StepId=841.0 request from UID:1000 </div>
<div dir="auto">GID:1000 HOST:192.168.1.1 PORT:38918</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: lllp_distribution: JobId=841 </div>
<div dir="auto">implicit auto binding: cores, dist 1</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: _task_layout_lllp_cyclic: </div>
<div dir="auto">_task_layout_lllp_cyclic</div>
<div dir="auto">[2021-02-01T23:58:19.789] task/affinity: _lllp_generate_cpu_bind: </div>
<div dir="auto">_lllp_generate_cpu_bind jobid [841]: mask_cpu, 0x000000000001000000000001</div>
<div dir="auto">[2021-02-01T23:58:19.825] [841.0] error: node10 [1] </div>
<div dir="auto">pmixp_client_v2.c:246 [pmixp_lib_init] mpi/pmix: ERROR: PMIx_server_init </div>
<div dir="auto">failed with error -2</div>
<div dir="auto">: Success (0)</div>
<div dir="auto">[2021-02-01T23:58:19.826] [841.0] error: node10 [1] pmixp_client.c:518 </div>
<div dir="auto">[pmixp_libpmix_init] mpi/pmix: ERROR: PMIx_server_init failed with error -1</div>
<div dir="auto">: Success (0)</div>
<div dir="auto">[2021-02-01T23:58:19.826] [841.0] error: node10 [1] pmixp_server.c:423 </div>
<div dir="auto">[pmixp_stepd_init] mpi/pmix: ERROR: pmixp_libpmix_init() failed</div>
<div dir="auto">[2021-02-01T23:58:19.826] [841.0] error: (null) [1] mpi_pmix.c:169 </div>
<div dir="auto">[p_mpi_hook_slurmstepd_prefork] mpi/pmix: ERROR: pmixp_stepd_init() failed</div>
<div dir="auto">[2021-02-01T23:58:19.829] [841.0] error: Failed mpi_hook_slurmstepd_prefork</div>
<div dir="auto">[2021-02-01T23:58:19.853] [841.0] error: job_manager exiting abnormally, </div>
<div dir="auto">rc = -1</div>
<div dir="auto">[2021-02-01T23:58:19.899] [841.0] done with job</div>
<div dir="auto"><br></div>
<div dir="auto">It seems that the culprit is the bind() failure, but I can't make much </div>
<div dir="auto">sense of it. I checked that /etc/hosts has everything correct and </div>
<div dir="auto">consistent with the info in slurm.conf.</div>
<div dir="auto"><br></div>
<div dir="auto">Other potentially relevant info: all compute nodes are diskless, they </div>
<div dir="auto">are pxe-booted from a NAS image and running ubuntu server 20.04. Running </div>
<div dir="auto">jobs on a single node is fine.</div>
<div dir="auto"><br></div>
<div dir="auto">Thanks for any insight and suggestions.</div>
<div dir="auto"><br></div>
<div dir="auto">Cheers,</div>
<div dir="auto">Andrej</div>
</blockquote>
</div><div dir="auto"><br></div>
</div></body>
</html>