[slurm-users] Rate Limiting of RPC calls

[Kevin Buckley]

On 2021/02/17 14:02, Kota Tsuyuzaki wrote:
>> .. so that I'm realizing that sacct may exhaust the resources more rapidly
> than squeue on mysql point of view because, as I understand correctly,
> squeue doesn't affects such a mysql db query performance.
> 
> Any thoughts?
> 
> Best,
> Kota


Pulling together some things from the various docs and man pages.


In the "Performance" section of the manual pages for sacct and squeue,
a clear disticntion is made between the two utilities, in that:

sacct  makes an RPC to slurmdbd

squeue makes an RPC to slurmctld

(although the diagram in the quickstart guide differs, in
  suggesting that sacct has links with all three of slurmctld,
  slurmdbd, and the slurmds ?)

but both man pages warn that too many invocations of either
utility can cause a degradation in the performance of their
respective daemon.

So yes, by replacing squeue with an sacct wrapper, you could
reduce the load on the slurmctld but degrade the performance
of the slurmdbd,

However, it's the slurmctld (the control daemon) that's doing
the real work, the scheduling of your jobs, whilst the slurmdbd,
and the database behind it, is merely storing accounting info
that the slurmctld sends it.

Furthermore, if the slurmdbd isn't responding to the slurmctld,
then the slurmctld will cache the accounting info until the
slurmdbd does respond.

However, if the slurmctld isn't doing its job, because it's
overloaded, then the throughput of whole of your cluster is
affected and the slurmctld won't have new accounting info to
send to the slurmdbd.

Ideally then, you don't want either slurmctld or slurmdbd
overloaded but, if you are forced to set things up so as to
make a choice, you would choose to send more of the load
towards the slurmdbd.

Kevin
-- 
Supercomputing Systems Administrator
Pawsey Supercomputing Centre