[slurm-users] srun jobfarming hassle question

[Bjørn-Helge Mevik]

"Ohlerich, Martin" <Martin.Ohlerich at lrz.de> writes:

> Dear Colleagues,
>
>
> already for quite some years now are we again and again facing issues on our clusters with so-called job-farming (or task-farming) concepts in Slurm jobs using srun. And it bothers me that we can hardly help users with requests in this regard.
>
>
> From the documentation (https://slurm.schedmd.com/srun.html#SECTION_EXAMPLES), it reads like this.
>
> ------------------------------------------->
>
> ...
>
> #SBATCH --nodes=??
>
> ...
>
> srun -N 1 -n 2 ... prog1 &> log.1 &
>
> srun -N 1 -n 1 ... prog2 &> log.2 &


Unfortunately, that part of the documentation is not quite up-to-date.
The semantics of srun has changed a little the last couple of
years/Slurm versions, so today, you have to use "srun --exact ...".  From
"man srun" (version 21.08):

       --exact
              Allow  a step access to only the resources requested for the
              step.  By default, all non-GRES resources on  each  node  in
              the  step  allocation will be used. This option only applies
              to step allocations.
              NOTE: Parallel steps will  either  be  blocked  or  rejected
              until  requested step resources are available unless --over‐
              lap is specified. Job resources can be held after  the  com‐
              pletion  of  an  srun  command while Slurm does job cleanup.
              Step epilogs and/or SPANK  plugins  can  further  delay  the
              release of step resources.

-- 
Regards,
Bjørn-Helge Mevik, dr. scient,
Department for Research Computing, University of Oslo

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230118/8820d02a/attachment-0001.sig>