If you are using AccountingStoreFlags=job_script and/or job_env in slurm.conf to store job scripts in the Slurm database, you may be interested in a highly useful feature introduced in Slurm 26.05: Previously, the stored job_scripts and job_env entries would accumulate without bounds in the database, potentially causing database performance or size issues, especially while upgrading Slurm. With 26.05 there are some new slurmdbd.conf options which can be used to purge job_script or job_env entries in the database, see ticket_23818 [1]. This new feature isn't mentioned at all in the Release notes [4], and is only inconspicuously mentioned way down in the Changes section [3] as:
Adding new archive/purge options to allow for explicit archiving of job_scripts and job_env without jobs.
The safe way to enable the new purge parameters is to introduce them very gradually, for example: PurgeJobScriptAfter=2000days PurgeJobEnvAfter=2000days and lower the values little by little over time (restarting slurmdbd each time). The MaxPurgeLimit [5] parameter may alternatively be used to limit the amount of database work. Best regards, Ole [1] https://support.schedmd.com/show_bug.cgi?id=23818 [2] https://slurm.schedmd.com/slurm.conf.html#OPT_AccountingStoreFlags [3] https://github.com/SchedMD/slurm/releases [4] https://slurm.schedmd.com/release_notes.html [5] https://slurm.schedmd.com/slurmdbd.conf.html#OPT_MaxPurgeLimit -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark