[slurm-users] MPI Spawn with IntelMPI and multiple nodes

[Craig Warren]

Hi folks,

I am trying (unsuccessfully) to use MPI Spawn with IntelMPI and Slurm over multiple nodes. A single node works OK using mpiexec.hydra:

mpiexec.hydra -n 1 python ./script_here.py

My MPI is a task farm essentially based on - https://github.com/jbornschein/mpi4py-examples/blob/master/10-task-pull-spawn.py. The full code is at - https://github.com/gprMax/gprMax/blob/master/gprMax/gprMax.py#L323

I have also tried using srun but it does not spawn the correct number of tasks.

Any advice on routes to solve the problem would be most welcome!

My full submit script:

#!/bin/bash

#SBATCH --account=****
#SBATCH --nodes=2
#SBATCH --ntasks=48
#SBATCH --cpus-per-task=1
#SBATCH --output=gprmax_mpi_cpu_2nodes-out.%j
#SBATCH --error=gprmax_mpi_cpu_2nodes-err.%j
#SBATCH --time=00:05:00
#SBATCH --partition=devel

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module --force purge
module use /usr/local/software/jureca/OtherStages
module load Stages/2018b
module load Intel IntelMPI

cd /p/project/****/****/gprMax
source activate gprMax

mpiexec.hydra -n 1 python -m gprMax user_models/cylinder_Bscan_2D.in -n 47 -mpi 48

Kind regards,

Craig

Dr Craig Warren CEng MIMechE MIET FHEA
Senior Lecturer, Department of Mechanical & Construction Engineering

[Northumbria University]

T: +44 (0)191 227 3633
E: craig.warren at northumbria.ac.uk<mailto:craig.warren at northumbria.ac.uk>
W: northumbria.ac.uk<http://www.northumbria.ac.uk>
Twitter: @DrCraigWarren<http://www.twitter.com/DrCraigWarren>, @enginerdsUK<http://www.twitter.com/enginerdsUK>

Room 117, Wynne Jones Building, Northumbria University, Newcastle upon Tyne, NE1 8ST, United Kingdom

This message is intended solely for the addressee and may contain confidential and/or legally privileged information. Any use, disclosure or reproduction without the sender's explicit consent is unauthorised and may be unlawful. If you have received this message in error, please notify Northumbria University immediately and permanently delete it. Any views or opinions expressed in this message are solely those of the author and do not necessarily represent those of the University. Northumbria University email is provided by Microsoft Office365 and is hosted within the EEA, although some information may be replicated globally for backup purposes. The University cannot guarantee that this message or any attachment is virus free or has not been intercepted and/or amended.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190430/3c7fb343/attachment.html>