[slurm-users] MPI jobs via mirun vs. srun through PMIx.

[Paul Edmon]

For my two-cents I would recommend using srun. While mpirun "works" I've 
seen strange behavior especially if you are using task affinity and core 
binding.  Even weirder with hybrid codes that use threads and MPI.  
Using srun resolves these issues as it integrates more tightly with the 
scheduler.  Sometime for older versions of MPI they need to use mpirun 
but by and large our community uses srun for the above reasons.  It's 
the more native slurm way of doing things with MPI.

-Paul Edmon-

On 9/17/19 4:12 AM, Marcus Wagner wrote:
> Hi Jürgen,
>
> we set in our modules the variables $MPIEXEC and $FLAGS_MPI_BATCH and 
> documented these.
> This way, by changing the workloadmanagement system or the MPI or 
> whatsoever does not change the documentation (at least on that point ;) )
>
> Best
> Marcus
>
>
> On 9/17/19 9:02 AM, Juergen Salk wrote:
>> * Loris Bennett <loris.bennett at fu-berlin.de> [190917 07:46]:
>>>>> But I still don't get the point. Why should I favour `srun 
>>>>> ./my_mpi_program´
>>>>> over `mpirun ./my_mpi_program´? For me, both seem to do exactly 
>>>>> the same
>>>>> thing. No? Did I miss something?
>>>>> Best regards
>>>>> Jürgen
>>>> Running a single job in isolation doesn't properly illustrate what 
>>>> slurm does in the larger sense.
>>>>
>>>> [...]
>>>>
>>> I think Jürgen may well be aware of what Slurm does in the larger 
>>> sense.
>> Dear Loris,
>>
>> thank you for your clarification. You are right. It seems that I have
>> phrased my question somewhat misleadingly.
>>
>> For our next cluster we will switch from Moab/Torque to Slurm and have
>> to adapt the documentation and example batch scripts for the users.
>> Therefore, I wonder if and why we should recommend (or maybe even urge)
>> our users to use srun instead of mpirun/mpiexec in their batch scripts
>> for MPI jobs.
>>
>>> I suspect the question, which I also have, is more like:
>>>
>>>    "What difference does it make whether I use 'srun' or 'mpirun' 
>>> within
>>>     a batch file started with 'sbatch'."
>> That's exactly the question I wanted to ask.
>>
>> Thanks again.
>>
>> Best regards
>> Jürgen
>>
>