[slurm-users] srun and Intel MPI 2020 Update 4

[Ciaron Linstead]

Hello all

I've been trying to run a simple MPI application (the Intel MPI 
Benchmark) using the latest Intel Parallel Studio (2020 Update 4) and 
srun. Version 2019 Update 4 runs this example correctly, as does mpirun.

SLURM is 17.11.7

The error I get is the following, unless I use --exclusive:


MPI startup(): Could not import some environment variables. Intel MPI 
process pinning will not be used.
                Possible reason: Using the Slurm srun command. In this 
case, Slurm pinning will be used.
MPIR_pmi_virtualization(): MPI startup(): PMI calls are forwarded to 
/p/system/slurm/lib/libpmi.so
Abort(2664079) on node 19 (rank 19 in comm 0): Fatal error in 
PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(136).........:
MPID_Init(1127)...............:
MPIDI_SHMI_mpi_init_hook(29)..:
MPIDI_POSIX_mpi_init_hook(141):
MPIDI_POSIX_eager_init(2109)..:
MPIDU_shm_seg_commit(296).....: unable to allocate shared memory



I have a ticket open with Intel, who suggested increasing /dev/shm on 
the nodes to 64GB (the size of the RAM on the nodes), but this had no 
effect.

Here's my submit script:



#!/bin/bash

#SBATCH --ntasks=25 # fails, unless exclusive
##SBATCH --exclusive

source 
/p/system/packages/intel/parallel_studio_xe_2020_update4/impi/2019.9.304/intel64/bin/mpivars.sh 
-ofi_internal=1

export I_MPI_FABRICS=shm:ofi
export FI_PROVIDER=verbs
export FI_VERBS_IFACE=ib0
export FI_LOG_LEVEL=trace

export I_MPI_PMI_LIBRARY=/p/system/slurm/lib/libpmi.so
export I_MPI_DEBUG=5

# Fails for any MPI program, not just this one
srun -v -n $SLURM_NTASKS /home/linstead/imb_2019.5/IMB-MPI1 barrier



Do you have any ideas about where/how to investigate this further?

Many thanks

Ciaron