[slurm-users] speed / efficiency of sacct vs. scontrol

[Loris Bennett]

Hi David,

David Laehnemann <david.laehnemann at hhu.de> writes:

[snip (16 lines)]

> P.S.: @Loris and @Noam: Exactly, snakemake is a software distinct from
> slurm that you can use to orchestrate large analysis workflows---on
> anything from a desktop or laptop computer to all kinds of cluster /
> cloud systems. In the case of Slurm it will submit each analysis step
> on a particular sample as a separate job, specifying the resources it
> needs. The scheduler then handles it from there. But because you can
> have (hundreds of) thousands of jobs, and with dependencies among them,
> you can't just submit everything all at once, but have to keep track of
> where you are at. And make sure you don't submit much more than the
> system can handle at any time, so you don't overwhelm the Slurm queue.

[snip (86 lines)]

I know what Snakemake and other workflow managers, such as Nextflow are
for, but my maybe ill-informed impression is that, while something of
this sort is obviously needed to manage complex dependencies, the
current solutions, probably because they originated outside the HPC
context, to try to do too much.  You say Snakemake helps

  make sure you don't submit much more than the system can handle

but that in my view should not be necessary.  Slurm has configuration
parameters which can be set to limit the number of jobs a user can
submit and/or run.  And when it comes to submitting (hundreds of)
thousands of jobs, Nextflow for example currently can't create job
arrays, and so generates large numbers of jobs with identical resource
requirements, which can prevent backfill from working properly.
Skimming the documentation for Snakemake, I also could not find any
reference to Slurm job arrays, so this could also be an issue.

Just my slightly grumpy 2¢.

Cheers,

Loris

-- 
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin