IIUC, when you suspend a job it remains in memory but with no CPU time allocated. If you reboot the node, the job state is lost (unless it uses checkpointing). When you restarted the jobs, they actually began a new run (Slurm doesn't know if they use checkpointing or not). You've been lucky that your jobs seems to use checkpointing...
The pocedure we're following when a node reboot is required is to create a reservation (or drain the nodes), leave jobs run until completion or time limit and when the nodes are free we reboot 'em.
Diego
Il 22/11/2024 00:05, Ratnasamy, Fritz via slurm-users ha scritto:
Hi,
I am using an old slurm version 20.11.8 and we had to reboot our cluster today for maintenance. I suspended all the jobs on it with the command scontrol suspend list_job_ids and all the jobs paused and were suspended. However, when I tried to resume them after the reboot, scontrol resume did not work (it was showing in the reason column " (JobHeldAdmin)". I was able to release them with scontrol release and the jobs started to run back. However, the SLURM recorded time on it resetted (Time columns, showing 0:00 for all the jobs) though the jobs seem to have re-started from the last point before he got suspended. 1- Did I follow the right procedure to suspend, reboot and resume/release? 2- In this case, does the wall time for all the jobs goes into reset and therefore anyone with slurm admin rights will be able to have their jobs last longer than the wall time limit by suspending and resuming a job?
Best,
*Fritz Ratnasamy*
Data Scientist
Information Technology