[slurm-users] MPI_Comm_spawn - doesn't work with sbatch, works with salloc

[Deij-van Rijswijk, Menno]

Good afternoon,

A simulation package I'm using is peforms MPI_Comm_spawn to dynamically spawn simulation processes. This works fine interactively on a head-node, but I'm running into problems when I try to submit a job to the SLURM scheduler using sbatch. If I use salloc instead, it does work. I would like to find out if I can also get to work with sbatch or that this is a known limitation.

I am using Intel compilers and MPI , version 2020 Update 1.

It can be reproduced with a small program that does this, the source code of which is given at the end of this message.

This works (when the $PWD is in $PATH):
salloc -N 2 mpiexec -np 1 spawn_example

This does not work
sbatch job

with job-file containing:
#================
#!/bin/bash
#SBATCH -N 2
#SBATCH --job-name=P12345.678

source /etc/profile.d/modules.sh

export I_MPI_DEBUG=500

mpiexec -np 1 ./spawn_example
#================

Further details: If I attach a gdb session to the process that performs the spawning, the following is found. It looks like it is hanging in MPI_Comm_spawn while trying to read something.

#0  0x00001555537ce8b2 in read () from /lib64/libpthread.so.0
#1  0x00001555544fd0ef in read (__fd=<optimized out>, __buf=<optimized out>, __nbytes=<optimized out>)
    at /usr/include/bits/unistd.h:45
#2  PMIU_readline (fd=5,
    buf=0x155554eedca0 "cmd=get_result rc=0 msg=success value=mpi#0200CD280A950027", '0' <repeats 16 times>, "$\n",
    maxlen=1023) at ../../src/pmi/simple/simple_pmiutil.c:143
#3  0x00001555544facde in VPMI_Spawn_multiple (count=5, cmds=0x155554eedca0, argvs=0x3ff,
    maxprocs=0x1555537ce8b2 <read+18>, info_keyval_sizes=0x0, info_keyval_vectors=0x28, preput_keyval_size=1,
    preput_keyval_vector=0x7f20000a0b40, errors=0x7f20001f1840) at ../../src/pmi/simple/simple_pmi.c:951
#4  0x00001555543a5f69 in MPIR_pmi_spawn_multiple (count=5, commands=0x155554eedca0, argvs=0x3ff,
    maxprocs=0x1555537ce8b2 <read+18>, info_ptrs=0x0, num_preput_keyval=40, preput_keyvals=0x28, pmi_errcodes=0x1)
    at ../../src/util/mpir_pmi.c:1417
#5  0x0000155553fdc85d in MPID_Comm_spawn_multiple (count=5, commands=0x155554eedca0, argvs=0x3ff,
    maxprocs=0x1555537ce8b2 <read+18>, info_ptrs=0x0, root=40, comm_ptr=0x0,
    intercomm=0x155554022e32 <PMPI_Comm_spawn+946>, errcodes=0x155554f52b60 <MPIR_Comm_builtin>)
    at ../../src/mpid/ch4/src/ch4_spawn.c:85
#6  0x0000155554022e32 in PMPI_Comm_spawn (command=0x7ffffffface0 "", argv=0x7fffffffadd4, maxprocs=0,
    info=1400694962, root=0, comm=40, intercomm=0x0, array_of_errcodes=0x7fffffffafd0)
    at ../../src/mpi/spawn/comm_spawn.c:118
#7  0x0000000000400c88 in main (argc=1, argv=0x7fffffffafd8, envp=0x7fffffffafe8) at spawn_example.c:53

================= source code =====================

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>

#define NUM_SPAWNS 56

int main(int argc, char* argv[])
{
int np = NUM_SPAWNS;
int errorcodes[NUM_SPAWNS];

MPI_Comm parentcomm, intercomm;

MPI_Init(&argc, &argv);
MPI_Comm_get_parent(&parentcomm);
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);

if (MPI_COMM_NULL == parentcomm)
{
MPI_Comm_spawn("spawn_example", MPI_ARGV_NULL, np, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, errorcodes);
printf("I'm the parent with rank=%d\n", rank);
}
else
{
printf("I'm spawned with rank %d\n", rank);
}
fflush(stdout);
MPI_Finalize();
return 0;
}

Thank you for your insight,

All the best,
Menno Deij - van Rijswijk


dr. ir. Menno A. Deij-van Rijswijk  | Researcher | Research & Development
MARIN | T +31 317 49 35 06 | mailto:M.Deij at marin.nl | http://www.marin.nl