[slurm-users] ntasks and cpus-per-task

[Paul Edmon]

At least from my experience wonky things can happen with slurm 
(especially if you have thread affinity on) if you don't rightly divide 
between -n and -c.  In general I've been telling our users that -c is 
for threaded applications and -n is for rank based parallelism.  This 
way the thread affinity works out properly.

-Paul Edmon-


On 02/22/2018 08:50 AM, Christopher Benjamin Coffey wrote:
> Loris,
>
> It’s simple, tell folks only to use -n for mpi jobs, and -c otherwise (default).
>
> It’s a big deal if folks use -n when it’s not an mpi program. This is because the non mpi program is launched n times (instead of once with internal threads) and will stomp over logs and output files (uncoordinated) leading to poor performance and incorrect results.
>
> Best,
> Chris
>> On Feb 22, 2018, at 1:52 AM, Loris Bennett <loris.bennett at fu-berlin.de> wrote:
>>
>> Hi Chris,
>>
>> Christopher Samuel <chris at csamuel.org> writes:
>>
>>>> On 22/02/18 18:49, Miguel Gutiérrez Páez wrote:
>>>>
>>>> What's the real meaning of ntasks? Has cpus-per-task and ntasks the
>>>> same meaning in sbatch and srun?
>>> --ntasks is for parallel distributed jobs, where you can run lots of
>>> independent processes that collaborate using some form of communication
>>> between the processes (usually MPI for HPC).
>>>
>>> So inside your batch script you would use "srun" to start up the tasks.
>>>
>>> However, unless you code is written to make use of that interface then
>>> it's not really going to help you, and so for any multithreaded
>>> application you need to use --cpus-per-task instead.
>> [snip (11 lines)]
>>
>> But does it make any difference for a multithreaded program if I have
>>
>>   #SBATCH --ntasks=4
>>   #SBATCH --nodes=1-1
>>
>> rather than
>>
>>   #SBATCH --ntasks=1
>>   #SBATCH --cpus-per-task=4
>>
>> Up to now I have only thought of --cpus-per-task in connection with
>> hybrid MPI/OpenMP jobs, which we don't actually have.  Thus I tend to
>> tell users to think always in terms of tasks, regardless of whether
>> these are MPI processes or just threads.
>>
>> One downside of my approach is that if the user forgets to specify
>> --nodes and --ntasks is greater than 1, non-MPI jobs can be assigned to
>> multiple nodes.
>>
>> Cheers,
>>
>> Loris
>>
>> -- 
>> Dr. Loris Bennett (Mr.)
>> ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de
>>