[slurm-users] OpenMPI interactive change in behavior?

[John DeSantis]

Hello all,

We've recently (don't laugh!) updated two of our SLURM installations from 16.05.10-2 to 20.11.3 and 17.11.9, respectively.  Now, OpenMPI doesn't seem to function in interactive mode across multiple nodes as it did previously on the latest version 20.11.3;  using `srun` and `mpirun` on a single node gives desired results, while using multiple nodes causes a hang. Jobs submitted via `sbatch` do _work as expected_.  

[desantis at sclogin0 ~]$ scontrol show config |grep VERSION; srun -n 2 -N 2-2 -t 00:05:00 --pty /bin/bash
SLURM_VERSION           = 17.11.9
[desantis at sccompute0 ~]$ for OPENMPI in mpi/openmpi/1.8.5 mpi/openmpi/2.0.4 mpi/openmpi/2.0.4-psm2 mpi/openmpi/2.1.6 mpi/openmpi/3.1.6 compilers/intel/2020_cluster_xe; do module load $OPENMPI ; which mpirun; mpirun hostname; module purge; echo; done
/apps/openmpi/1.8.5/bin/mpirun
sccompute0
sccompute1

/apps/openmpi/2.0.4/bin/mpirun
sccompute1
sccompute0

/apps/openmpi/2.0.4-psm2/bin/mpirun
sccompute1
sccompute0

/apps/openmpi/2.1.6/bin/mpirun
sccompute0
sccompute1

/apps/openmpi/3.1.6/bin/mpirun
sccompute0
sccompute1

/apps/intel/2020_u2/compilers_and_libraries_2020.2.254/linux/mpi/intel64/bin/mpirun
sccompute1
sccompute0


15:58:28 Mon Apr 26 <0>
desantis at itn0
[~] $ scontrol show config|grep VERSION; srun -n 2 -N 2-2 --qos=devel --partition=devel -t 00:05:00 --pty /bin/bash
SLURM_VERSION           = 20.11.3
srun: job 1019599 queued and waiting for resources
srun: job 1019599 has been allocated resources
15:58:46 Mon Apr 26 <0>
desantis at mdc-1057-30-1
[~] $ for OPENMPI in mpi/openmpi/1.8.5 mpi/openmpi/2.0.4 mpi/openmpi/2.0.4-psm2 mpi/openmpi/2.1.6 mpi/openmpi/3.1.6 compilers/intel/2020_cluster_xe; do module load $OPENMPI ; which mpirun; mpirun hostname; module purge; echo; done
/apps/openmpi/1.8.5/bin/mpirun
^C
/apps/openmpi/2.0.4/bin/mpirun
^C
/apps/openmpi/2.0.4-psm2/bin/mpirun
^C
/apps/openmpi/2.1.6/bin/mpirun
^C
/apps/openmpi/3.1.6/bin/mpirun
^C
/apps/intel/2020_u2/compilers_and_libraries_2020.2.254/linux/mpi/intel64/bin/mpirun
^C[mpiexec at mdc-1057-30-1] Sending Ctrl-C to processes as requested
[mpiexec at mdc-1057-30-1] Press Ctrl-C again to force abort
^C

Our SLURM installations are fairly straight forward.  We `rpmbuild` directly from the bzip2 files without any additional arguments.  We've done this since we first started using SLURM with version 14.03.3-2 and through all upgrades.  Due to SLURM's awesomeness(!), we've simply used the same configuration files between version changes, with the only changes being made to parameters which have been deprecated/renamed.  Our "Mpi{Default,Params}" have always been sent to "none".  The only real difference we're able to ascertain is that the MPI plugin for openmpi has been removed.

svc-3024-5-2: SLURM_VERSION           = 16.05.10-2
svc-3024-5-2: srun: MPI types are...
svc-3024-5-2: srun: mpi/openmpi
svc-3024-5-2: srun: mpi/mpich1_shmem
svc-3024-5-2: srun: mpi/mpichgm
svc-3024-5-2: srun: mpi/mvapich
svc-3024-5-2: srun: mpi/mpich1_p4
svc-3024-5-2: srun: mpi/lam
svc-3024-5-2: srun: mpi/none
svc-3024-5-2: srun: mpi/mpichmx
svc-3024-5-2: srun: mpi/pmi2

viking: SLURM_VERSION           = 20.11.3
viking: srun: MPI types are...
viking: srun: cray_shasta
viking: srun: pmi2
viking: srun: none

sclogin0: SLURM_VERSION           = 17.11.9
sclogin0: srun: MPI types are...
sclogin0: srun: openmpi
sclogin0: srun: none
sclogin0: srun: pmi2
sclogin0: 

As far as building OpenMPI, we've always withheld any SLURM specific flags, i.e. "--with-slurm", although during the build process SLURM is detected.  

Because OpenMPI was always built using this method, we never had to recompile OpenMPI after subsequent SLURM upgrades, and no cluster ready applications had to be rebuilt.  The only time OpenMPI had to be rebuilt was due to OPA hardware which was a simple addition of the "--with-psm2" flag.

It is my understanding that the openmpi plugin "never really did anything" (per perusing the mailing list), which is why it was removed.  Furthermore, searching the mailing list suggests that the appropriate method is to use `salloc` first, despite version 17.11.9 not needing `salloc` for an "interactive" sessions.

Before we go further down this rabbit hole, were other sites affected with a transition from SLURM versions 16.x,17.x,18.x(?) to versions 20.x?  If so, did the methodology for multinode interactive MPI sessions change?

Thanks!
John DeSantis