[slurm-users] Rate Limiting of RPC calls

[Kota Tsuyuzaki]

Hello guys, 


> > 1) It should be possible to write a wrapper, or even a binary,
> >     that gives the user the squeue format using the API calls
> >     which sacct targets, for a subset of squeue functionality ?
> >
> > 2) How much of the functionality of squeue would be lost if
> >     SchedMD had only provided an "sacct with squeue formatting"
> >     and how much of the lost functionality would really be
> >     missed?
> >

Thinking of use of sacct instead of squeue, I got interesting results on resource usage viewpoint.
At first, exactly sacct communicates slurmdbd directly. It means it'll reduce the communication to slurmctld.
However, when I ran sacct pure queries (e.g. just `sacct` command) resulting in bunch of job records more than hundreds, mysql db raised CPU usage. Looking at query log in mysql db, it looks like slurmdbd requested many similar "select" queries for each RPC execution. The difference between the select queries for one `sacct` execution looks only job_db_inx so it seems something (slurmdbd? mysql subquery?) would request more select queries according to the number of records. And more, when I ran sacct with only one job id (i.e. `sacct -j <jobid>`), the mysql CPU spike was reduced so that I'm realizing that sacct may exhaust the resources more rapidly than squeue on mysql point of view because, as I understand correctly, squeue doesn't affects such a mysql db query performance. 

Any thoughts?

Best,
Kota

--------------------------------------------
露崎　浩太 (Kota Tsuyuzaki)
kota.tsuyuzaki.pc at hco.ntt.co.jp
NTTソフトウェアイノベーションセンタ
分散処理基盤技術プロジェクト
0422-59-2837
---------------------------------------------

> -----Original Message-----
> From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Kota Tsuyuzaki
> Sent: Friday, February 12, 2021 6:14 PM
> To: 'Slurm User Community List' <slurm-users at lists.schedmd.com>; 'Christopher Samuel' <chris at csamuel.org>
> Subject: Re: [slurm-users] Rate Limiting of RPC calls
> 
> Thanks Guys!
> 
> All information is valuable. I'll look up our setting and try to tune our Slurm cluster to get higher performance.
> 
> Best,
> Kota
> 
> --------------------------------------------
> 露崎　浩太 (Kota Tsuyuzaki)
> kota.tsuyuzaki.pc at hco.ntt.co.jp
> NTTソフトウェアイノベーションセンタ
> 分散処理基盤技術プロジェクト
> 0422-59-2837
> ---------------------------------------------
> 
> > -----Original Message-----
> > From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of
> > Kevin Buckley
> > Sent: Wednesday, February 10, 2021 11:32 AM
> > To: Slurm User Community List <slurm-users at lists.schedmd.com>;
> > Christopher Samuel <chris at csamuel.org>
> > Subject: Re: [slurm-users] Rate Limiting of RPC calls
> >
> > On 2021/02/10 09:33, Christopher Samuel wrote:
> > >
> > > Also getting users to use `sacct` rather than `squeue` to check what
> > > state a job is in can help a lot too, it reduces the load on slurmctld.
> >
> > That raises an interesting take on the two utilities, Chris, in that
> >
> > 1) It should be possible to write a wrapper, or even a binary,
> >     that gives the user the squeue format using the API calls
> >     which sacct targets, for a subset of squeue functionality ?
> >
> > 2) How much of the functionality of squeue would be lost if
> >     SchedMD had only provided an "sacct with squeue formatting"
> >     and how much of the lost functionality would really be
> >     missed?
> >
> > Kevin
> > --
> > Supercomputing Systems Administrator
> > Pawsey Supercomputing Centre
>