[slurm-users] speed / efficiency of sacct vs. scontrol

[Sean Maxwell]

Hi David,

Those queries then should not have to happen too often, although do you
> have any indication of a range for when you say "you still wouldn't
> want to query the status too frequently." Because I don't really, and
> would probably opt for some compromise of every 30 seconds or so.
>

Every 30 seconds sounds reasonable. My cautioning was only in the sense
that everything has limitations. For example, the query processing time is
dependent on the size of the query and the overall load on the system, so
any static interval you select may not work well under some conditions. You
might want to defend against that by making the interval adaptive, like the
maximum of 30s or 5x the execution time of the last query, so that it
adapts to the overall burden of the query and the system load. That is just
an example to try and communicate what I was getting at.


> One thing I didn't understand from your eMail is the part about job
> names, as the command I gave doesn't use job names for its query:
>
> sacct -X -P -n --format=JobIdRaw,State -j <jobid_1>,<jobid_2>,...
>
> Instead, it just uses the JobId, and isn't that guaranteed to be unique
> at any point in time? Or were you meaning to say that JobId can be non-
> unique? That would indeed spell trouble on a different level, and make
> status checks much more complicated...
>

Job id is unique. What I mean is, building a CSV list of jobs has
scalability issues. If you could assign the same job name to each job in
the snakemake pipeline, then the query is much shorter, and still returns
the status for each job id that snakemake has launched. Rather than falling
back to scontrol (which doesn't support querying by name) snakemake could
fall back to squeue which does support querying by name.

Best,

-Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230224/b14aa4da/attachment.htm>