[slurm-users] speed / efficiency of sacct vs. scontrol

[David Laehnemann]

Hi Sean,

yes, this is exactly what snakemake currently does. I didn't write that
code, but from my previous debugging, I think handling one job at a
time was simply the logic of the general executor for cluster systems,
and makes things like querying via scontrol as a fallback easier to
handle. But this is not set in stone.

But from your comment I understand that handling these queries in
batches would be less work for slurmdbd, right? So instead of querying
each jobid with a separate database query, it would do one database
query for the whole list? Is that really easier for the system, or
would it end up doing a call for each jobid, anyway?

And just to be as clear as possible, a call to sacct would then look
like this:
sacct -X -P -n --format=JobIdRaw,State -j <jobid_1>,<jobid_2>,...

Because we can of course rewrite the respective code section, so any
insight on how to do this job accounting more efficiently (and better
tailored to how Slurm does things) is appreciated.

cheers,
david


On Thu, 2023-02-23 at 09:46 -0500, Sean Maxwell wrote:
> Hi David,
> 
> On Thu, Feb 23, 2023 at 8:51 AM David Laehnemann <
> david.laehnemann at hhu.de>
> wrote:
> 
> > Quick follow-up question: do you have any indication of the rate of
> > job
> > status checks via sacct that slurmdbd will gracefully handle (per
> > second)? Or any suggestions how to roughly determine such a rate
> > for a
> > given cluster system?
> > 
> 
> I looked at your PR for context, and this line of snakemake looks
> problematic (I know this isn't part of your PR, it is part of the
> original
> code)
> https://github.com/snakemake/snakemake/commit/a0f04bab08113196fe1616a621bd6bf20fc05688#diff-d1b47826c1fc35806df72508e2f5e7f1d0424f9b2f7b9124810b051f5fe97f1bL296
> :
> 
> sacct_cmd = f"sacct -P -n --format=JobIdRaw,State -j {jobid}"
> 
> Since jobid is an int, this looks like snakmake will individually
> probe
> each Slurm job it has launched. If snakemake was using batch logic to
> gather status for all your running jobs with one call to sacct, then
> you
> could probably set the interval low. But it looks like it is going to
> probe
> each job individually by ID, so it will make as many RPC calls as
> their are
> jobs in the pipeline when it is time to check the status.
> 
> I could be wrong, but this is how I evaluated the code without going
> farther upstream.
> 
> Best,
> 
> -Sean