Hi All,
I'm managing a cluster with Slurm, consisting of 4 nodes. One of the compute nodes appears to be experiencing issues. While the front node's 'squeue' command indicates that jobs are running, upon connecting to the problematic node, I observe no active
processes and GPUs are not being utilized.
[sushil@ccbrc ~]$ sinfo -Nel
Wed May 29 12:00:08 2024
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
gag 1 defq* mixed 48 2:24:1 370000 0 1 (null) none
gag 1 glycore mixed 48 2:24:1 370000 0 1 (null) none
glyco1 1 defq* completing* 128 2:64:1 500000 0 1 (null) none
glyco1 1 glycore completing* 128 2:64:1 500000 0 1 (null) none
glyco2 1 defq* mixed 128 2:64:1 500000 0 1 (null) none
glyco2 1 glycore mixed 128 2:64:1 500000 0 1 (null) none
mannose 1 defq* mixed 24 2:12:1 180000 0 1 (null) none
mannose 1 glycore mixed 24 2:12:1 180000 0 1 (null) none
On glyco1 (affected node!):
squeue # gets stuck
sudo systemctl restart slurmd # gets stuck
I tried the following to clear the jobs stuck in CG state, but any new job appears to be stuck in a 'running' state without actually running.
scontrol update nodename=glyco1 state=down reason=cg
scontrol update nodename=glyco1 state=resume reason=cg
There is no I/O issue in that node, and all file systems are under 30% in use. Any advice on how to resolve this without rebooting the machine?
Best,
Sushil