[slurm-users] Slurm Node Unresponsive
grant.campbell at mythic-ai.com
Tue Sep 8 14:10:09 UTC 2020
I am running a Slurm cluster that I inherited from an employee who left, so
you will have to forgive any ignorance on my part, I am still coming up to
speed on some core concepts.
I have a vexing issue where one slurm node becomes unresponsive
consistently. Network and DNS seem to be working fine, but the control node
logs "Nodes node3 not responding, setting DOWN ". If I mark the node as
RESUME it comes back up, but no jobs can be scheduled, I have to restart
the slurmd process to get it to work.
I enabled debug logging on the troublesome node, and I see it logging
errors like the below near constantly:
[2020-09-08T09:02:35.189] [59921.0] error: Unable to establish controller
[2020-09-08T09:02:40.584] [59924.0] error: Unable to establish controller
[2020-09-08T09:03:02.550] [59923.extern] error: Unable to establish
[2020-09-08T09:03:04.537] [59921.extern] error: Unable to establish
[2020-09-08T09:03:09.474] [59924.extern] error: Unable to establish
This of course seems problematic, though it should be noted I do not see
the logging of these errors correlate with the outage chronologically at
all -- as I said, they log near constantly.
One final piece of context, this machine OOM'd last week, and this issue
began after we brought it back up. As part of that process, I had to
re-join it to IPA, so not sure if there is something there that could have
caused this issue.
Any help or advice would be much appreciated, thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the slurm-users