[slurm-users] batched and efficient job status queries by snakemake using sacct

[David Laehnemann]

Hi again, everybody,

based on the feedback here on the mailing list and on GitHub, and lots
of digging into the docs, I have now changed snakemake's behaviour to
do much fewer status queries and to do them in database-optimised
batches via sacct. As there is a lot of user and admin knowledge around
here, I wondered if any of you might find the time to comment on the
suggested changes. These are in the open pull request here, with the
most important infos in the initial comment starting the pull request:

https://github.com/snakemake/snakemake/pull/2167

We're aiming to merge and release this in the next couple of days, to
get this out quickly and enable users to reduce strain on slurm
clusters as soon as possible (by updating to the then latest version of
snakemake). So if it's already merged by the time you manage to have a
look, nevertheless don't hesitate to comment on the pull request,
comment here or open a new issue on the snakemake repository (optimally
pinging me in) -- we'll gladly further improve the situation, here.

Also, in case the solutions provided for snakemake job status queries
find approval here, it might make sense to point developers of other
workflow managers to this pull request, whenever you find such workflow
management systems overstraining your slurm setup.

Otherwise, many thanks already to Tim Wickberg, Sean Maxwell, Bas van
der Vlies and Brian Andrus for the particularly helpful feedback and
suggestions that led to the current solution!

cheers,
david