[slurm-users] Node/job failures following scontrol reconfigure command
D.J.Baker at soton.ac.uk
Thu Oct 4 05:19:27 MDT 2018
We have just finished an upgrade to slurm 18.08. My last task was to reset the slurmctld/slurmd timeouts to sensible values -- as they were set prior to the update. That is..
SlurmctldTimeout = 60 sec
SlurmdTimeout = 300 sec
With slurm <18.08 I've reconfigure the cluster many times before without an issues. Yesterday I found that this commands "pushed" most of the compute nodes into a "NODE_FAIL" state resulting in the loss of most running jobs.
I'm wondering if anyone has seen anything like this on their cluster, and if so what the solution was. I would be interested in hearing your experiences, please. Maybe I need to revise/increase the timeout values -- this sort of issue is tricky to test on an active cluster
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the slurm-users