My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed.
The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure?
Douglas O'Neal, Ph.D. (contractor) Manager, HPC Systems Administration Group, ITOG Frederick National Laboratory for Cancer Research Leidos Biomedical Research, Inc. Phone: 301-228-4656 Email: Douglas.O'Neal@nih.govmailto:Doug%20O'Neal%20%3cDouglas.O'Neal@nih.gov%3e
Instead of using the archive files, couldn't you query the db directly for the info you need?
I would recommend sacct/sreport if those can get the info you need.
Brian Andrus
On 5/28/2024 9:59 AM, O'Neal, Doug (NIH/NCI) [C] via slurm-users wrote:
My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed.
The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure?
Douglas O’Neal, Ph.D. (contractor)
Manager, HPC Systems Administration Group, ITOG
Frederick National Laboratory for Cancer Research
Leidos Biomedical Research, Inc.
Phone: 301-228-4656
Email: Douglas.O’Neal@nih.gov mailto:Doug%20O'Neal%20%3cDouglas.O'Neal@nih.gov%3e
Oh, to address the passed train:
Restore the archive data with "sacctmgr archive load", then you can do as you need.
From man sacctmgr:
*archive*{dump|load} </SPECS/>
Write database information to a flat file or load information that has previously been written to a file.
Brian Andrus
Setup your other MariaDB instance, dump the current slurmdbd and restore/import it, then restore your archive
On 5/28/2024 11:38 AM, Brian Andrus wrote:
Instead of using the archive files, couldn't you query the db directly for the info you need?
I would recommend sacct/sreport if those can get the info you need.
Brian Andrus
On 5/28/2024 9:59 AM, O'Neal, Doug (NIH/NCI) [C] via slurm-users wrote:
My organization needs to access historic job information records for metric reporting and resource forecasting. slurmdbd is archiving only the job information, which should be sufficient for our numbers, but is using the default archive script. In retrospect, this data should have been migrated to a secondary MariaDB instance, but that train has passed.
The format of the archive files is not well documented. Does anyone have a program (python/C/whatever) that will read a job_table_archive file and decode it into a parsable structure?
Douglas O’Neal, Ph.D. (contractor)
Manager, HPC Systems Administration Group, ITOG
Frederick National Laboratory for Cancer Research
Leidos Biomedical Research, Inc.
Phone: 301-228-4656
Email: Douglas.O’Neal@nih.gov mailto:Doug%20O'Neal%20%3cDouglas.O'Neal@nih.gov%3e
-
Brian Andrus -
O'Neal, Doug (NIH/NCI) [C]