[slurm-users] Intel MPI startup

[Peter Kjellström]

On Mon, 29 Apr 2019 15:47:49 +0000
Michael Robbert <mrobbert at mines.edu> wrote:

> I was curious what startup method other sites are using with Intel
> MPI?

We (nsc.liu.se nat. level academic HPC) have used IntelMPI+slurm as a
primary platform for several generations.

Our setup to this day is "mpiexec.hydra -bootstrap slurm".

Pros:
 * it works well
 * it binds well including with slurm cgroups
 * it performs well

For those of you not familiar with the approach it basically goes like
this:

site wrapper -> mpirun -> mpiexec.hydra -> srun pmi_proxy

That is, the intel pmi_proxy ends up under the slurmstepd and then
launches and pinns the ranks/threads.

When I say it performs well I include experiences up to configs such as
3200 ranks with 16 threads each and 64000 ranks pure MPI.

Disclaimer: IntelMPI-2019 is "problematic" and not yet successfully
used here...

/Peter K