Hi All,
I'm managing a cluster with Slurm, consisting of 4 nodes. One of the compute nodes appears to be experiencing issues. While the front node's 'squeue' command indicates that jobs are running, upon connecting to the problematic node, I observe no active processes and GPUs are not being utilized.
[sushil@ccbrc ~]$ sinfo -Nel Wed May 29 12:00:08 2024 NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON gag 1 defq* mixed 48 2:24:1 370000 0 1 (null) none gag 1 glycore mixed 48 2:24:1 370000 0 1 (null) none glyco1 1 defq* completing* 128 2:64:1 500000 0 1 (null) none glyco1 1 glycore completing* 128 2:64:1 500000 0 1 (null) none glyco2 1 defq* mixed 128 2:64:1 500000 0 1 (null) none glyco2 1 glycore mixed 128 2:64:1 500000 0 1 (null) none mannose 1 defq* mixed 24 2:12:1 180000 0 1 (null) none mannose 1 glycore mixed 24 2:12:1 180000 0 1 (null) none
On glyco1 (affected node!): squeue # gets stuck sudo systemctl restart slurmd # gets stuck
I tried the following to clear the jobs stuck in CG state, but any new job appears to be stuck in a 'running' state without actually running. scontrol update nodename=glyco1 state=down reason=cg scontrol update nodename=glyco1 state=resume reason=cg
There is no I/O issue in that node, and all file systems are under 30% in use. Any advice on how to resolve this without rebooting the machine?
Best, Sushil