_node_config_validate: gres/gpu: Count changed on node (0 != 2)
Hey Slurm-users list, while our regular gpu nodes are working fine, our on demand gpu nodes have a weird issue. They power up, I can ssh onto them and execute nvidia-smi on them without issue, but they are marked invalid and slurmctld logs _node_config_validate: gres/gpu: Count changed on node (0 != 2) however, node show shows that the gpus are recognized and the gres.conf are stored on the worker nodes as expected and the node entries in the slurm.conf are fine, too: # slurm.conf NodeName=my_worker_node SocketsPerBoard=16 CoresPerSocket=1 RealMemory=64075 MemSpecLimit=4000 State=CLOUD Gres=gpu:L4:2 # openstack # gres.conf on my_worker_node ubuntu@my_node:~$ cat /etc/slurm/gres.conf # GRES CONFIG Name=gpu Type=L4 File=/dev/nvidia0 Name=gpu Type=L4 File=/dev/nvidia1 Thankful for any ideas and debugging ideas. Best, Xaver PS: By executing: sudo scontrol update NodeName=$(bibiname 0) Gres= sudo scontrol reconfigure sudo scontrol update NodeName=$(bibiname 0) state=RESUME reason=None the node can be resumed. However, this is not a real solution.
Hey Slurm-users list, meanwhile I was able to find: [2026-03-23T12:58:16.105] debug: gres/gpu: init: loaded [2026-03-23T12:58:16.105] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-03-23T12:58:16.105] warning: Ignoring file-less GPU gpu:L4 from final GRES list [2026-03-23T12:58:16.105] debug: skipping GRES for NodeName=my_worker_node Name=gpu File=/dev/nv dia0 [2026-03-23T12:58:16.105] debug: skipping GRES for NodeName=my_worker_node Name=gpu File=/dev/nv dia1 [2026-03-23T12:58:16.105] debug: gres/gpu: init: loaded [2026-03-23T12:58:16.106] debug: skipping GRES for NodeName=my_worker_node Name=gpu File=/dev/nv dia0 [2026-03-23T12:58:16.106] debug: skipping GRES for NodeName=my_worker_node Name=gpu File=/dev/nv dia1 [2026-03-23T12:58:16.106] debug: gres/gpu: init: loaded so I am wondering whether that is the issue. I also noticed that after powering up the node without requesting a gpu (works), scheduling to the node by requesting a GPU is not an issue. Best, Xaver On 3/20/26 17:21, Xaver Stiensmeier wrote:
Hey Slurm-users list,
while our regular gpu nodes are working fine, our on demand gpu nodes have a weird issue. They power up, I can ssh onto them and execute nvidia-smi on them without issue, but they are marked invalid and slurmctld logs
_node_config_validate: gres/gpu: Count changed on node (0 != 2)
however, node show shows that the gpus are recognized and the gres.conf are stored on the worker nodes as expected and the node entries in the slurm.conf are fine, too:
# slurm.conf NodeName=my_worker_node SocketsPerBoard=16 CoresPerSocket=1 RealMemory=64075 MemSpecLimit=4000 State=CLOUD Gres=gpu:L4:2 # openstack
# gres.conf on my_worker_node ubuntu@my_node:~$ cat /etc/slurm/gres.conf # GRES CONFIG Name=gpu Type=L4 File=/dev/nvidia0 Name=gpu Type=L4 File=/dev/nvidia1
Thankful for any ideas and debugging ideas.
Best, Xaver
PS: By executing:
sudo scontrol update NodeName=$(bibiname 0) Gres= sudo scontrol reconfigure sudo scontrol update NodeName=$(bibiname 0) state=RESUME reason=None
the node can be resumed. However, this is not a real solution.
Hi everyone, On 3/23/26 14:11, Xaver Stiensmeier via slurm-users wrote: [...]
so I am wondering whether that is the issue. I also noticed that after powering up the node without requesting a gpu (works), scheduling to the node by requesting a GPU is not an issue. [...]
We noticed this as well: after powering up a node the GPU device-files (/dev/nvidia*) are not created (immediately). What we did: we changed the slurmd.service file and added ExecStartPre=-/path/to/nvidia-smi -L to the [Service] section. This creates the device files and a failure (e.g. on non-GPU nodes) is ignored by systemd (due to the "-" before the command). Maybe this helps? Kind regards, Hermann
Hey, I am not 100% sure yet as that needs further testing (in case it is a race condition), but I think I was able to fix my issue by using the NodeName gres.conf format and supplying it to each node instead of placing the gres.conf files just on the nodes with gres with the node specific information. Best, Xaver On 3/23/26 15:13, Hermann Schwärzler via slurm-users wrote:
Hi everyone,
On 3/23/26 14:11, Xaver Stiensmeier via slurm-users wrote: [...]
so I am wondering whether that is the issue. I also noticed that after powering up the node without requesting a gpu (works), scheduling to the node by requesting a GPU is not an issue. [...]
We noticed this as well: after powering up a node the GPU device-files (/dev/nvidia*) are not created (immediately).
What we did: we changed the slurmd.service file and added
ExecStartPre=-/path/to/nvidia-smi -L
to the [Service] section. This creates the device files and a failure (e.g. on non-GPU nodes) is ignored by systemd (due to the "-" before the command).
Maybe this helps?
Kind regards, Hermann
participants (2)
-
Hermann Schwärzler -
Xaver Stiensmeier