[slurm-users] Unexpected MPI process distribution with the --exclusive flag

[CB]

Hi Everyone,

I've recently discovered that when an MPI job is submitted with the
--exclusive flag, Slurm fills up each node even if the --ntasks-per-node
flag is used to set how many MPI processes is scheduled on each node.
 Without the --exclusive flag, Slurm works fine as expected.

Our system is running with Slurm 17.11.7.

The following options works that each node has 16 MPI processes until all
980 MPI processes are scheduled.with total of 62 compute nodes.  Each of
the 61 nodes has 16 MPI processes and the last one has 4 MPI processes,
which is 980 MPI processes in total.
#SBATCH -n 980
#SBATCH --ntasks-per-node=16

However, if the --exclusive option is added, Slurm fills up each node with
28 MPI processes (the compute node has 28 cores).  Interestingly, Slurm
still allocates  62 compute nodes although  only 35 nodes of them are
actually used to distribute 980 MPI processes.

#SBATCH -n 980
#SBATCH --ntasks-per-node=16
#SBATCH --exclusive

Has anyone seen this behavior?

Thanks,
- Chansup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190730/9c814bbd/attachment.htm>