[slurm-users] Question about sbatch options: -n, and --cpus-per-task

[Stephen Cousins]

If you want to have the same number of processes per node, like:

#PBS -l nodes=4:ppn=8

then what I am doing (maybe there is another way?) is:

#SBATCH --ntasks-per-node=8
#SBATCH --nodes=4
#SBATCH --mincpus=8

This is because "--ntasks-per-node" is actually "maximum number of tasks
per node" and "--nodes=4" means "minimum number of nodes". I'm sure other
variations (specifying --ntasks=32, --mincpus=8 and --nodes=4-4 might do it
too) but this one is what I've been using. I remember being surprised when
coming over from Torque to find that "--ntasks-per-node" and --nodes did
not mean what they so obviously seemed to mean.


Steve

On Thu, Mar 24, 2022 at 7:56 PM David Henkemeyer <david.henkemeyer at gmail.com>
wrote:

> Thank you!  We recently converted from pbs, and I was converting “ppn=X”
> to “-n X”.  Does it make more sense to convert “ppn=X” to
> --“cpus-per-task=X”?
>
> Thanks again
> David
>
> On Thu, Mar 24, 2022 at 3:54 PM Thomas M. Payerle <payerle at umd.edu> wrote:
>
>> Although all three cases ( "-N 1 --cpus-per-task 64 -n 1", "-N 1
>> --cpus-per-task 1 -n 64", and "-N 1 --cpus-per-task 32 -n 2") will cause
>> Slurm to allocate 64 cores to the job, there can (and will) be differences
>> in the other respects.
>>
>> The variable SLURM_NTASKS will be set to the argument of the -n (aka
>> --ntasks) argument, and other Slurm variables will differ as well.
>>
>> More importantly, as others noted, srun will launch $SLURM_NTASKS
>> processes.  The mpirun/mpiexec/etc binaries of most MPI libraries will (if
>> compiled with support for Slurm) act similarly (and indeed, I believe most
>> use srun under the hood).
>>
>> If you are just using sbatch and launching a single process using 64
>> threads, then the different options are probably equivalent for most intent
>> and purposes.  Similar if you are doing a loop to start 64 single threaded
>> processes.  But those are simplistic cases, and just happen to "work" even
>> though you are "abusing" the scheduler options.  And even the cases wherein
>> it "works" is subject to unexpected failures (e.g. if one substitutes srun
>> for sbatch).
>>
>> The differences are most clear when the -N 1 flag is not given.
>> Generally, SLURM_NTASKS should be the number of MPI or similar tasks you
>> intend to start.  By default, it is assumed the tasks can support
>> distributed memory parallelism, so the scheduler by default assumes that it
>> can launch tasks on different nodes (the -N 1 flag you mentioned would
>> override that).  Each such task is assumed to need --cpus-per-task cores
>> which the scheduler assumes needs shared memory parallelism (i.e. must be
>> on the same node).
>> So without the -N 1, "--cpus-per-task 64 -n 1" will require 64 cores on a
>> single node, whereas "-n 64 --cpus-per-task 1" can result in the job being
>> assigned 64 cores on a single node to a single core on 64 nodes or any
>> combination in between with 64 cores.  The "--cpus-per-task 32 -n 2" will
>> either assign one node with 64 cores or 2 nodes with 32 cores each.
>>
>> As I said, although there are some simple cases where the different cases
>> are mostly functionally equivalent, I would recommend trying to use the
>> proper arguments --- "abusing" the arguments might work for a while but
>> will likely bite you in the end.  E.g., the 64 thread case should do
>> "--cpus-per-task 64", and the launching processes in the loop should
>> _probably_ do "-n 64" (assuming it can handle the tasks being assigned to
>> different nodes).
>>
>> On Thu, Mar 24, 2022 at 3:35 PM David Henkemeyer <
>> david.henkemeyer at gmail.com> wrote:
>>
>>> Assuming -N is 1 (meaning, this job needs only one node), then is there
>>> a difference between any of these 3 flag combinations:
>>>
>>> -n 64 (leaving cpus-per-task to be the default of 1)
>>> --cpus-per-task  64 (leaving -n to be the default of 1)
>>> --cpus-per-task 32 -n 2
>>>
>>> As far as I can tell, there is no functional difference. But if there is
>>> even a subtle difference, I would love to know what it is!
>>>
>>> Thanks
>>> David
>>> --
>>> Sent from Gmail Mobile
>>>
>>
>>
>> --
>> Tom Payerle
>> DIT-ACIGS/Mid-Atlantic Crossroads        payerle at umd.edu
>> 5825 University Research Park               (301) 405-6135
>> University of Maryland
>> College Park, MD 20740-3831
>>
> --
> Sent from Gmail Mobile
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220324/b52f3b5a/attachment-0001.htm>