[slurm-users] good practices

[Loris Bennett]

Hi Nigella,

Nigella Sanders <nigella.sanders at gmail.com> writes:

> Thank you all for such interesting replies.
>
> The --dependency option is quite useful but in practice it has some
> inconvenients. Firstly, all 20 jobs are instantly queued which some
> users may be interpreting as an abusive use of common resources.

This doesn't seem a problem to me, since no common resources are being
used by jobs in the queue.  It only becomes a problem if a single person
can queue enough jobs to consume all the resources *and* you are not using
any form of fairshare.  Otherwise job started later, but with a higher
priority will start earlier, if the resources become available.

This is not to say that users might *think* that a large number of jobs
belonging other users automatically means that later jobs will be
disadvantages.  However, that is more an issue of educating your users.

> Even worse, if a job fails, the rest one will stay queued forever (?)
> being the first tagged as "DependencyNeverSatisfied", and the rest
> just as "Dependency".

This is just a consequence of your requirement that "each job ... needs
the previous one to be completed", but it also isn't a problem, because
pending jobs don't consume resources for which users complete.

Regards

Loris

> PS: Yarom, with queue time I meant the total run time allowed. I my case, after a job starts running it will be killed if it takes more than 10 hours of execution time. If the partition queue time limit were of 10 days
> for instance I guess I could use a single sbatch to launch an script containing the 20 executions as steps with srun
>
> Regards,
> Nigella
>
> El lun., 25 nov. 2019 a las 15:08, Yair Yarom (<irush at cs.huji.ac.il>) escribió:
>
>  Hi,
>
>  I'm not sure what queue time limit of 10 hours is. If you can't have jobs waiting for more than 10 hours, than it seems to be very small for 8 hours jobs.
>  Generally, a few options:
>  a. The --dependency option (either afterok or singleton)
>  b. The --array option of sbatch with limit of 1 job at a time (instead of the for loop): sbatch --array=1-20%1 
>  c. At the end of the script of each job, call the sbatch line of the next job (this is probably the only option if indeed I understood the queue time limit correctly).
>
>  And indeed, srun should probably be reserved for strictly interactive jobs.
>
>  Regards,
>      Yair.
>
>  On Mon, Nov 25, 2019 at 11:21 AM Nigella Sanders <nigella.sanders at gmail.com> wrote:
>
>  Hi all,
>
>  I guess this is a simple matter but I still find it confusing.
>
>  I have to run 20 jobs on our supercomputer. 
>  Each job takes about 8 hours and every one need the previous one to be completed.
>  The queue time limit for jobs is 10 hours.
>
>  So my first approach is serially launching them in a loop using srun:
>
>  #!/bin/bash
>  for i in {1..20};do
>      srun  --time 08:10:00  [options]
>  done
>
>  However SLURM literature keeps saying that 'srun' should be only used for short command line tests. So that some sysadmins would consider this a bad practice (see this).
>
>  My second approach switched to sbatch:
>
>  #!/bin/bash 
>  for i in {1..20};do
>      sbatch  --time 08:10:00 [options]
>      [polling to queue to see if job is done]
>  done
>
>  But since sbatch returns the prompt I had to add code to check for job termination. Polling make use of sleep command and it is prone to race conditions so it doesn't like to sysadmins either.
>
>  I guess there must be a --wait option in some recent versions of SLURM (see this). Not yet available in our system though.
>
>  Is there any prefererable/canonical/friendly way to do this?
>  Any thoughts would be really appreciated,
>
>  Regards,
>  Nigella.
>
-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de