Maybe a somewhat 'hacky' idea - couldn't you put a line in the epilog script that logs the interesting entries (i.e. SLURM_SUBMIT_HOST) to some logfile at job completion? Of course that would only be feasible if the amount of completing jobs per time unit isn't super high, otherwise you'd obviously need to watch out for race conditions in concurrent writes, etc.
- René Sitt
Am 07.08.24 um 15:29 schrieb Steffen Grunewald via slurm-users:
On Wed, 2024-08-07 at 08:55:21 -0400, Slurm users wrote:
Warning on that one, it can eat up a ton of database space (depending on size of environment, uniqueness of environment between jobs, and number of jobs). We had it on and it nearly ran us out of space on our database host. That said the data can be really useful depending on the situation.
-Paul Edmon-
On 8/7/2024 8:51 AM, Juergen Salk via slurm-users wrote:
Hi Steffen,
not sure if this is what you are looking for, but with `AccountingStoreFlags=job_env´ set in slurm.conf, the batch job environment will be stored in the accounting database and can later be retrieved with `sacct -j <jobid> --env-vars´ command.
On Wed, 2024-08-07 at 14:56:30 +0200, Slurm users wrote:
What you're looking for might be doable simply by setting the AccountStoreFlags parameter in slurm.conf. [1]
Be aware, though, that job_env has sometimes been reported to grow quite large.
I see, I cannot have the cake and eat it at the same time. Given the size of our users' typical env, I'm dropping the idea for now - maybe this will come up again in the not-so-far future. (Maybe it's worth a feature request?)
Thanks everyone!
- Steffen