Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Thursday, October 15, 2026 6:40 PM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either. -Dj On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file. Another command that would be good to see is 'nvidia-smi topo -m'. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>