[slurm-users] MPI jobs via mirun vs. srun through PMIx.

[Stijn De Weirdt]

hi jurgen,

> For our next cluster we will switch from Moab/Torque to Slurm and have
> to adapt the documentation and example batch scripts for the users.
heh, we did that a year ago, and we made (well, fixed the slurm one) a
qsub wrapper to avoid having to document this and retraining our users.
(we do not have the luxury to assume our users are experts in the tools,
and to be honest, srun and sbatch are powerful but
complicated/confusing). we were lucky enough that the amount of features
we need to support is limited, so that's ok. we more or less migrated
live-ish (6 clusters, one at the time), we had little fallout from the
users. users still use qsub to submit jobs ;)


> Therefore, I wonder if and why we should recommend (or maybe even urge) 
> our users to use srun instead of mpirun/mpiexec in their batch scripts 
> for MPI jobs. 
we are going through this exercise at the moment, and main reason is for
simplicity.
mpirun != mpirun (at least long ago, nowadays it seems better), good
luck documenting that for user who don't fully comprehend what a CLI is.
for that reason we are maintaining another wrapper, and are currently
converting the wrapper to use srun, it makes things a lot easier in the
wrapper code.

but srun also seems not for the faint of heart, and i'm haivig serious
issues with gpus (but i'm not sure the mpiruns do any better in that
sense). (thankfully, our user will not need to know any of this stuff ;)

good luck

stijn

> 
>> I suspect the question, which I also have, is more like:
>>
>>   "What difference does it make whether I use 'srun' or 'mpirun' within
>>    a batch file started with 'sbatch'."
> 
> That's exactly the question I wanted to ask.
> 
> Thanks again.
> 
> Best regards
> Jürgen
>