[slurm-users] SlurmDB Archive settings?

[Paul Edmon]

We just use the Archive function built into slurm.  That has worked fine 
for us for the past 6 years.  We keep 6 months of data in the active 
archive.


If you have 6 years worth of data and you want to prune down to 2 years, 
I recommend going month by month rather than doing it in one go.  When 
we initially started archiving data several years back our first pass at 
archiving (which at that time had 2 years of data in it) took forever 
and actually caused issues with the archive process.  We worked with 
SchedMD, improved the archive script built into Slurm but also decided 
to only archive one month at a time which allowed it to get done in a 
reasonable amount of time.


The archived data can be pulled into a different slurm database, which 
is what we do for importing historic data into our XDMod instance.


-Paul Edmon-


On 7/13/2022 4:55 PM, Timony, Mick wrote:
> Hi Slurm Users,
>
> Currently we don't archive our SlurmDB and have 6 years' worth of data 
> in our SlurmDB. We are looking to start archiving our database as it 
> starting to get rather large, and we have decided to keep 2 years' 
> worth of data. I'm wondering what approaches or scripts other groups use.
>
> The docs refer to the ArchiveScript setting at:
> https://slurm.schedmd.com/slurmdbd.conf.html#OPT_ArchiveScript
>
> I've seen suggestions to import into another database that will 
> require keeping the schema up-to-date which seems like a possible 
> maintenance issue or nightmare if one forgets to update the schema 
> after updating Slurmdb. We also have most of the information in an 
> Elasticsearch <https://slurm.schedmd.com/elasticsearch.html> instance, 
> which will likely suite our needs for long term historical information.
>
>
> What do you use to archive this information? CSV files, SQL dumps or 
> something else?
>
>
> Regards
> -- 
> Mick Timony
> Senior DevOps Engineer
> Harvard Medical School
> --
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220714/696e5112/attachment.htm>