[slurm-users] Re: _node_config_validate: gres/gpu: Count changed on node (0 != 2)

Hi everyone, On 3/23/26 14:11, Xaver Stiensmeier via slurm-users wrote: [...]

so I am wondering whether that is the issue. I also noticed that after powering up the node without requesting a gpu (works), scheduling to the node by requesting a GPU is not an issue. [...]

We noticed this as well: after powering up a node the GPU device-files (/dev/nvidia*) are not created (immediately). What we did: we changed the slurmd.service file and added ExecStartPre=-/path/to/nvidia-smi -L to the [Service] section. This creates the device files and a failure (e.g. on non-GPU nodes) is ignored by systemd (due to the "-" before the command). Maybe this helps? Kind regards, Hermann