[slurm-users] Slurmdbd purge settings

[Luke Sudbery]

Command in question is:


sreport --parsable2 user topusage topcount=3 start=10/15/19 end=10/16/19

Similar to this: https://bugs.schedmd.com/show_bug.cgi?id=2315 where the problem eventually just 'went away'. We also have >12000 associations and see a large number of them (>9000) listed in the SQL query. Running the query directly without them in and it completes in a few seconds.

Cheers,

Luke

--
Luke Sudbery
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don't work on Monday.

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Luke Sudbery
Sent: 23 February 2021 12:26
To: slurm-users at lists.schedmd.com
Subject: [slurm-users] Slurmdbd purge settings

We have suddenly got bad performance from sreport, querying a 1 hour period (in the last 24 hours) for TopUsage went from taking under a minute to timing out after the 15 minutes max slurmdbd query time - although the SQL query on the DB server continued long after that.

So firstly we were wondering what might have caused that.

But while investigating we decided we should turn on purging records in slurmdbd.conf, and wanted more detail about when the purge would occur and would it lock the database for other Slurm processes. Docs say "The purge takes place at the start of the each purge interval." But we assume it will also do so on a restart of slurmdbd so we can manage exactly when that happens - is that true? And as we have many years and millions of records to purge we need to know if this will hang all database access, and what kind of outage that is likely to cause.

Anyone have experience of enabling urging after the fact?

Many thanks,

Luke

--
Luke Sudbery
Architecture, Infrastructure and Systems
Advanced Research Computing, IT Services
Room 132, Computer Centre G5, Elms Road

Please note I don't work on Monday.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210223/51247054/attachment.htm>