Re: Jobs not getting assigned the requested GPU card
Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> *Sent:* Thursday, October 15, 2026 1:42 PM *To:* slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment" Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file. Another command that would be good to see is 'nvidia-smi topo -m'. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either. -Dj On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. //dev/nvidia1/) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:02 PM *To:* Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.
We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.
As you surmised, the minor does not match the ID:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1
Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X
I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.
-Dj
On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Sent:* Thursday, October 15, 2026 1:42 PM *To:* slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list -- slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Thursday, October 15, 2026 6:40 PM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either. -Dj On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file. Another command that would be good to see is 'nvidia-smi topo -m'. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
It appears that slurmd -G aligns with the devices shown by nvidia-smi -L: # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34 -Dj On 10/15/26 20:45, Brian Haymore wrote:
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:40 PM *To:* Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either.
-Dj
On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html <https://slurm.schedmd.com/gres.html> it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. //dev/nvidia1/) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:02 PM *To:* Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.
We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.
As you surmised, the minor does not match the ID:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1
Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X
I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.
-Dj
On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Sent:* Thursday, October 15, 2026 1:42 PM *To:* slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list -- slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>
The -C is the more important one for what I'm asking I think. It is showing that it was able to detect the GPUs with nvml. Wanted to make sure your slurm build & env had what was needed. When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Thursday, October 15, 2026 6:53 PM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card It appears that slurmd -G aligns with the devices shown by nvidia-smi -L: # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34 -Dj On 10/15/26 20:45, Brian Haymore wrote: Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:40 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either. -Dj On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file. Another command that would be good to see is 'nvidia-smi topo -m'. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
Thanks, Brian. Trying the NVML autodetect approach again today. sinfo is showing the node status as "invalid" after changing to autodetect. slurmctld.log is complaining about socket boundaries: [2026-10-16T10:52:08.390] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T10:52:08.390] error: _slurm_rpc_node_registration node=moose66: Invalid argument Adding "Parameters=l3cache_as_socket" to the node config does not correct this issue. Node /etc/slurm/gres.conf: AutoDetect=nvml Node entry in slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=l3cache_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 # /opt/slurm/sbin/slurmd -G [2026-10-16T10:54:40.584] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T10:54:40.687] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:53:39 -Dj On 10/15/26 20:57, Brian Haymore wrote:
The -C is the more important one for what I'm asking I think. It is showing that it was able to detect the GPUs with nvml. Wanted to make sure your slurm build & env had what was needed.
When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:53 PM *To:* Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card It appears that slurmd -G aligns with the devices shown by nvidia-smi -L:
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
# /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34
-Dj
On 10/15/26 20:45, Brian Haymore wrote:
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:40 PM *To:* Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either.
-Dj
On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html <https://slurm.schedmd.com/gres.html> it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. //dev/nvidia1/) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> *Sent:* Thursday, October 15, 2026 6:02 PM *To:* Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.
We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.
As you surmised, the minor does not match the ID:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1
Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X
I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.
-Dj
On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:* Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Sent:* Thursday, October 15, 2026 1:42 PM *To:* slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:* [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list -- slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>
I'll hazard a guess that the cpu on moose66 is an AMD? I've seen this issue on a Naples and Rome systems with GPUs. I had to use this to get it to play nice on the AMD boxes (due to NUMA topology of the AMD chiplet design, verifiable with `numactl --hardware`, and I'm using AutoDetect=nvml
SlurmdParameters=numa_node_as_socket
I do not profess to be an expert on this, but I do know that the _merge_system_gres_conf <https://github.com/SchedMD/slurm/blob/slurm-25.11/src/plugins/gres/gpu/gres_gpu.c#:~:text=func-,_merge_system_gres_conf,-func> funciton is the one you're fighting with as it pertains to the naming of the gres. If you throw a few verbose flags at slurmd you can get some good debug output detailing what the nvml autodetect is seeing and doing.
slurmd -C -vvv slurmd -G -vv
I've never had issues with named substrings
Found gpu:tesla_p40:6,gpu:quadro_rtx_6000:2 with Autodetect=nvml (Substring of gpu name may be used instead)
[2026-10-16T12:33:03.711] Gres GPU plugin: Final merged GRES list: [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:-1,0,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-9c8c4abb-4099-e2e5-11c6-f290d767a767 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,-1,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-72e391a7-be28-b300-8a4b-ea8469531950 [2026-10-16T12:33:03.711] GRES[gpu] Type:rtx Count:1 Cores(88):0-21 Links:0,0,-1,2,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-e0ef1ca0-f749-8e4f-b9f7-a36d90f8b27e [2026-10-16T12:33:03.711] GRES[gpu] Type:rtx Count:1 Cores(88):0-21 Links:0,0,2,-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-fc568a4e-c008-e414-63af-52d29d4ba511 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-b4c05798-be9e-b1d7-e1ff-d6f9a8e64ddf [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia5 UniqueId:GPU-1eddadc4-edb3-99ba-16d6-0382df255265 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia6 UniqueId:GPU-e7f4838f-5986-4d13-6aa3-4e54bdec082a [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia7 UniqueId:GPU-f002a9a2-119f-a54e-f6ff-db9a15d411d7 [2026-10-16T12:33:03.711] GRES[shard] Type:(null) Count:48 Cores(88):(null) Links:(null) Flags: File:(null) UniqueId:(null)
For instance p40 and rtx (defined in slurm.conf) are both substrings of tesla_p40 and quadro_rtx_6000 respectively. I'm not sure if `rtx5090` would "match" `nvidia_geforce_rtx_5090`. I suspect `rtx_5090` would, if `rtx5090` did not, and I feel more sure that `5090` would match. If I had some idle systems to test against, I'd love to prove or disprove this theory. Hope this is at least somewhat helpful, Reed
On Oct 16, 2026, at 10:03 AM, Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> wrote:
Thanks, Brian. Trying the NVML autodetect approach again today.
sinfo is showing the node status as "invalid" after changing to autodetect.
slurmctld.log is complaining about socket boundaries:
[2026-10-16T10:52:08.390] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T10:52:08.390] error: _slurm_rpc_node_registration node=moose66: Invalid argument
Adding "Parameters=l3cache_as_socket" to the node config does not correct this issue.
Node /etc/slurm/gres.conf: AutoDetect=nvml
Node entry in slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=l3cache_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
# /opt/slurm/sbin/slurmd -G [2026-10-16T10:54:40.584] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T10:54:40.687] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
# /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:53:39
-Dj
On 10/15/26 20:57, Brian Haymore wrote:
The -C is the more important one for what I'm asking I think. It is showing that it was able to detect the GPUs with nvml. Wanted to make sure your slurm build & env had what was needed.
When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C From: Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:53 PM To: Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
It appears that slurmd -G aligns with the devices shown by nvidia-smi -L:
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
# /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34
-Dj
On 10/15/26 20:45, Brian Haymore wrote:
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C From: Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:40 PM To: Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either.
-Dj
On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C From: Dj Merrill <slurm@deej.net> <mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.
We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.
As you surmised, the minor does not match the ID:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1
Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X
I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.
-Dj
On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card
We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list -- slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi Reed! Thanks for your input! The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz". I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error: From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T13:06:29.573] error: _slurm_rpc_node_registration node=moose66: Invalid argument Here is the more verbose output you asked for: # /opt/slurm/sbin/slurmd -G -vv [2026-10-16T13:12:47.129] debug: Log file re-opened [2026-10-16T13:12:47.130] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2026-10-16T13:12:47.130] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T13:12:47.130] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2026-10-16T13:12:47.131] debug: auth/munge: init: loaded [2026-10-16T13:12:47.131] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2026-10-16T13:12:47.131] debug: tls/none: init: tls/none loaded [2026-10-16T13:12:47.135] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:12:47.135] GRES: Global AutoDetect=nvml(1) [2026-10-16T13:12:47.135] debug: gres/gpu: init: loaded [2026-10-16T13:12:47.135] debug: gres/shard: init: loaded [2026-10-16T13:12:47.136] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:12:47.206] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:12:47.211] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:12:47.217] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:12:47.224] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:12:47.226] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:12:47.231] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:12:47.232] Gres GPU plugin: Merging configured GRES with system GPUs [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] Gres GPU plugin: Final merged GRES list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] gres/shard: gres_c_s_init_share_devices: GRES: Initialized gres.conf list [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: shard device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: shard device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: shard device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: shard device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: shard device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] Final gres.conf list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.241] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin # /opt/slurm/sbin/slurmd -C -vvv [2026-10-16T13:13:23.840] debug2: hwloc_topology_init [2026-10-16T13:13:23.840] debug2: hwloc_topology_load [2026-10-16T13:13:23.844] debug2: _check_full_access: got full access to the cpuset topology [2026-10-16T13:13:23.844] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:13:23.844] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:13:23.851] debug2: gpu/nvml: _nvml_init: Successfully initialized NVML [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 13 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 16 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 5 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_5090 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:26:0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1A:00.0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0,0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 [2026-10-16T13:13:23.913] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.917] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.917] debug2: ------------------------------- [2026-10-16T13:13:23.917] debug2: *14001 MHz [0] [2026-10-16T13:13:23.917] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.917] debug2: --------------------------------- [2026-10-16T13:13:23.917] debug2: *3090 MHz [0] [2026-10-16T13:13:23.917] debug2: *3082 MHz [1] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *1635 MHz [194] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *187 MHz [387] [2026-10-16T13:13:23.917] debug2: *180 MHz [388] [2026-10-16T13:13:23.917] debug2: *13801 MHz [1] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *7001 MHz [2] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *810 MHz [3] [2026-10-16T13:13:23.919] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.919] debug2: --------------------------------- [2026-10-16T13:13:23.919] debug2: *3090 MHz [0] [2026-10-16T13:13:23.919] debug2: *3082 MHz [1] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *1635 MHz [194] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *187 MHz [387] [2026-10-16T13:13:23.919] debug2: *180 MHz [388] [2026-10-16T13:13:23.919] debug2: *405 MHz [4] [2026-10-16T13:13:23.920] debug2: Possible GPU Graphics Frequencies (95): [2026-10-16T13:13:23.920] debug2: --------------------------------- [2026-10-16T13:13:23.920] debug2: *885 MHz [0] [2026-10-16T13:13:23.920] debug2: *877 MHz [1] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *532 MHz [47] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *187 MHz [93] [2026-10-16T13:13:23.920] debug2: *180 MHz [94] [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:27:0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1B:00.0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0,0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 [2026-10-16T13:13:23.921] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.925] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.925] debug2: ------------------------------- [2026-10-16T13:13:23.925] debug2: *9501 MHz [0] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *9251 MHz [1] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *5001 MHz [2] [2026-10-16T13:13:23.926] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.926] debug2: --------------------------------- [2026-10-16T13:13:23.926] debug2: *2100 MHz [0] [2026-10-16T13:13:23.926] debug2: *2085 MHz [1] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *1155 MHz [63] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *225 MHz [125] [2026-10-16T13:13:23.926] debug2: *210 MHz [126] [2026-10-16T13:13:23.926] debug2: *810 MHz [3] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *2100 MHz [0] [2026-10-16T13:13:23.927] debug2: *2085 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *1155 MHz [63] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [125] [2026-10-16T13:13:23.927] debug2: *210 MHz [126] [2026-10-16T13:13:23.927] debug2: *405 MHz [4] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *420 MHz [0] [2026-10-16T13:13:23.927] debug2: *405 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *315 MHz [7] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [13] [2026-10-16T13:13:23.927] debug2: *210 MHz [14] [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:30:0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1E:00.0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,0,0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia4 [2026-10-16T13:13:23.928] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.932] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.932] debug2: ------------------------------- [2026-10-16T13:13:23.932] debug2: *9501 MHz [0] [2026-10-16T13:13:23.932] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.932] debug2: --------------------------------- [2026-10-16T13:13:23.932] debug2: *2100 MHz [0] [2026-10-16T13:13:23.932] debug2: *2085 MHz [1] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *1155 MHz [63] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *225 MHz [125] [2026-10-16T13:13:23.932] debug2: *210 MHz [126] [2026-10-16T13:13:23.932] debug2: *9251 MHz [1] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *5001 MHz [2] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *810 MHz [3] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *2100 MHz [0] [2026-10-16T13:13:23.934] debug2: *2085 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *1155 MHz [63] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [125] [2026-10-16T13:13:23.934] debug2: *210 MHz [126] [2026-10-16T13:13:23.934] debug2: *405 MHz [4] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *420 MHz [0] [2026-10-16T13:13:23.934] debug2: *405 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *315 MHz [7] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [13] [2026-10-16T13:13:23.934] debug2: *210 MHz [14] [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100_80gb_pcie [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:63:0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:3F:00.0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,-1,0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 [2026-10-16T13:13:23.936] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.937] debug2: Possible GPU Memory Frequencies (1): [2026-10-16T13:13:23.937] debug2: ------------------------------- [2026-10-16T13:13:23.937] debug2: *1512 MHz [0] [2026-10-16T13:13:23.938] debug2: Possible GPU Graphics Frequencies (81): [2026-10-16T13:13:23.938] debug2: --------------------------------- [2026-10-16T13:13:23.938] debug2: *1410 MHz [0] [2026-10-16T13:13:23.938] debug2: *1395 MHz [1] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *810 MHz [40] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *225 MHz [79] [2026-10-16T13:13:23.938] debug2: *210 MHz [80] [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 4: [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:64:0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:40:00.0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,0,-1 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 [2026-10-16T13:13:23.938] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.942] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.942] debug2: ------------------------------- [2026-10-16T13:13:23.942] debug2: *9501 MHz [0] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *9251 MHz [1] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *5001 MHz [2] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *810 MHz [3] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *405 MHz [4] [2026-10-16T13:13:23.945] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.945] debug2: --------------------------------- [2026-10-16T13:13:23.945] debug2: *420 MHz [0] [2026-10-16T13:13:23.945] debug2: *405 MHz [1] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *315 MHz [7] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *225 MHz [13] [2026-10-16T13:13:23.945] debug2: *210 MHz [14] [2026-10-16T13:13:23.945] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:13:23.954] debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML [2026-10-16T13:13:23.954] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin [2026-10-16T13:13:23.955] Configured with rsmi, but rsmi isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] Configured with oneAPI, but oneAPI isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] debug: gpu/nrt: init: init: GPU NRT plugin loaded [2026-10-16T13:13:23.955] error: System GPU detection failed [2026-10-16T13:13:23.955] debug: gpu/nrt: fini: fini: unloading GPU NRT plugin NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-03:12:06 -Dj On 10/16/26 12:45, Reed Dier wrote:
I'll hazard a guess that the cpu on moose66 is an AMD? I've seen this issue on a Naples and Rome systems with GPUs.
I had to use this to get it to play nice on the AMD boxes (due to NUMA topology of the AMD chiplet design, verifiable with `numactl --hardware`, and I'm using AutoDetect=nvml
SlurmdParameters=numa_node_as_socket
I do not profess to be an expert on this, but I do know that the _merge_system_gres_conf <https://github.com/SchedMD/slurm/blob/slurm-25.11/src/plugins/gres/gpu/gres_gpu.c#:~:text=func-,_merge_system_gres_conf,-func> funciton is the one you're fighting with as it pertains to the naming of the gres.
If you throw a few verbose flags at slurmd you can get some good debug output detailing what the nvml autodetect is seeing and doing.
slurmd -C -vvv slurmd -G -vv
I've never had issues with named substrings
Found gpu:tesla_p40:6,gpu:quadro_rtx_6000:2 with Autodetect=nvml (Substring of gpu name may be used instead)
[2026-10-16T12:33:03.711] Gres GPU plugin: Final merged GRES list: [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:-1,0,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-9c8c4abb-4099-e2e5-11c6-f290d767a767 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,-1,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-72e391a7-be28-b300-8a4b-ea8469531950 [2026-10-16T12:33:03.711] GRES[gpu] Type:rtx Count:1 Cores(88):0-21 Links:0,0,-1,2,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-e0ef1ca0-f749-8e4f-b9f7-a36d90f8b27e [2026-10-16T12:33:03.711] GRES[gpu] Type:rtx Count:1 Cores(88):0-21 Links:0,0,2,-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-fc568a4e-c008-e414-63af-52d29d4ba511 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-b4c05798-be9e-b1d7-e1ff-d6f9a8e64ddf [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia5 UniqueId:GPU-1eddadc4-edb3-99ba-16d6-0382df255265 [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia6 UniqueId:GPU-e7f4838f-5986-4d13-6aa3-4e54bdec082a [2026-10-16T12:33:03.711] GRES[gpu] Type:p40 Count:1 Cores(88):0-21 Links:0,0,0,0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia7 UniqueId:GPU-f002a9a2-119f-a54e-f6ff-db9a15d411d7 [2026-10-16T12:33:03.711] GRES[shard] Type:(null) Count:48 Cores(88):(null) Links:(null) Flags: File:(null) UniqueId:(null)
For instance p40 and rtx (defined in slurm.conf) are both substrings of tesla_p40 and quadro_rtx_6000 respectively.
I'm not sure if `rtx5090` would "match" `nvidia_geforce_rtx_5090`. I suspect `rtx_5090` would, if `rtx5090` did not, and I feel more sure that `5090` would match.
If I had some idle systems to test against, I'd love to prove or disprove this theory.
Hope this is at least somewhat helpful, Reed
On Oct 16, 2026, at 10:03 AM, Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> wrote:
Thanks, Brian. Trying the NVML autodetect approach again today.
sinfo is showing the node status as "invalid" after changing to autodetect.
slurmctld.log is complaining about socket boundaries:
[2026-10-16T10:52:08.390] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T10:52:08.390] error: _slurm_rpc_node_registration node=moose66: Invalid argument
Adding "Parameters=l3cache_as_socket" to the node config does not correct this issue.
Node /etc/slurm/gres.conf: AutoDetect=nvml
Node entry in slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=l3cache_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
# /opt/slurm/sbin/slurmd -G [2026-10-16T10:54:40.584] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T10:54:40.687] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
# /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:53:39
-Dj
On 10/15/26 20:57, Brian Haymore wrote:
The -C is the more important one for what I'm asking I think. It is showing that it was able to detect the GPUs with nvml. Wanted to make sure your slurm build & env had what was needed.
When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:*Dj Merrill<slurm@deej.net> *Sent:*Thursday, October 15, 2026 6:53 PM *To:*Brian Haymore<brian.haymore@utah.edu>; slurm-users<slurm-users@lists.schedmd.com> *Subject:*Re: [slurm-users] Jobs not getting assigned the requested GPU card It appears that slurmd -G aligns with the devices shown by nvidia-smi -L:
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
# /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34
-Dj
On 10/15/26 20:45, Brian Haymore wrote:
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show?
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:*Dj Merrill<slurm@deej.net> <mailto:slurm@deej.net> *Sent:*Thursday, October 15, 2026 6:40 PM *To:*Brian Haymore<brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users<slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:*Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either.
-Dj
On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over onhttps://slurm.schedmd.com/gres.html <https://slurm.schedmd.com/gres.html> it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g.//dev/nvidia1/) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:*Dj Merrill<slurm@deej.net> <mailto:slurm@deej.net> *Sent:*Thursday, October 15, 2026 6:02 PM *To:*Brian Haymore<brian.haymore@utah.edu> <mailto:brian.haymore@utah.edu>; slurm-users<slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:*Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.
We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.
As you surmised, the minor does not match the ID:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1
Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4
# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0
# nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)
# nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X
I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.
-Dj
On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.
Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done
If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.
Another command that would be good to see is 'nvidia-smi topo -m'.
-- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C <http://bit.ly/1HO1N2C> ------------------------------------------------------------------------ *From:*Dj Merrill via slurm-users<slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Sent:*Thursday, October 15, 2026 1:42 PM *To:*slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com><slurm-users@lists.schedmd.com> <mailto:slurm-users@lists.schedmd.com> *Subject:*[slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo.
Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour.
We have a node with 5 GPU cards, of three types:
% nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)
/etc/slurm/gres.conf matches the above output (and has been this way for years):
% cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4
The /etc/slurm/slurm.conf entry for this node is the same as it has been for years:
NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700
When submitting a job such as:
$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources
$ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
Any suggestions as to where to look?
Thank you,
-Dj
-- slurm-users mailing list --slurm-users@lists.schedmd.com <mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email toslurm-users-leave@lists.schedmd.com <mailto:slurm-users-leave@lists.schedmd.com>
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi Reed! Thanks for your input! The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz". I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error: From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T13:06:29.573] error: _slurm_rpc_node_registration node=moose66: Invalid argument Here is the more verbose output you asked for: # /opt/slurm/sbin/slurmd -G -vv [2026-10-16T13:12:47.129] debug: Log file re-opened [2026-10-16T13:12:47.130] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2026-10-16T13:12:47.130] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T13:12:47.130] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2026-10-16T13:12:47.131] debug: auth/munge: init: loaded [2026-10-16T13:12:47.131] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2026-10-16T13:12:47.131] debug: tls/none: init: tls/none loaded [2026-10-16T13:12:47.135] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:12:47.135] GRES: Global AutoDetect=nvml(1) [2026-10-16T13:12:47.135] debug: gres/gpu: init: loaded [2026-10-16T13:12:47.135] debug: gres/shard: init: loaded [2026-10-16T13:12:47.136] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:12:47.206] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:12:47.211] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:12:47.217] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:12:47.224] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:12:47.226] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:12:47.231] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:12:47.232] Gres GPU plugin: Merging configured GRES with system GPUs [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] Gres GPU plugin: Final merged GRES list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] gres/shard: gres_c_s_init_share_devices: GRES: Initialized gres.conf list [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: shard device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: shard device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: shard device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: shard device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: shard device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] Final gres.conf list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.241] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin # /opt/slurm/sbin/slurmd -C -vvv [2026-10-16T13:13:23.840] debug2: hwloc_topology_init [2026-10-16T13:13:23.840] debug2: hwloc_topology_load [2026-10-16T13:13:23.844] debug2: _check_full_access: got full access to the cpuset topology [2026-10-16T13:13:23.844] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:13:23.844] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:13:23.851] debug2: gpu/nvml: _nvml_init: Successfully initialized NVML [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 13 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 16 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 5 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_5090 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:26:0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1A:00.0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0,0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 [2026-10-16T13:13:23.913] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.917] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.917] debug2: ------------------------------- [2026-10-16T13:13:23.917] debug2: *14001 MHz [0] [2026-10-16T13:13:23.917] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.917] debug2: --------------------------------- [2026-10-16T13:13:23.917] debug2: *3090 MHz [0] [2026-10-16T13:13:23.917] debug2: *3082 MHz [1] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *1635 MHz [194] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *187 MHz [387] [2026-10-16T13:13:23.917] debug2: *180 MHz [388] [2026-10-16T13:13:23.917] debug2: *13801 MHz [1] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *7001 MHz [2] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *810 MHz [3] [2026-10-16T13:13:23.919] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.919] debug2: --------------------------------- [2026-10-16T13:13:23.919] debug2: *3090 MHz [0] [2026-10-16T13:13:23.919] debug2: *3082 MHz [1] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *1635 MHz [194] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *187 MHz [387] [2026-10-16T13:13:23.919] debug2: *180 MHz [388] [2026-10-16T13:13:23.919] debug2: *405 MHz [4] [2026-10-16T13:13:23.920] debug2: Possible GPU Graphics Frequencies (95): [2026-10-16T13:13:23.920] debug2: --------------------------------- [2026-10-16T13:13:23.920] debug2: *885 MHz [0] [2026-10-16T13:13:23.920] debug2: *877 MHz [1] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *532 MHz [47] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *187 MHz [93] [2026-10-16T13:13:23.920] debug2: *180 MHz [94] [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:27:0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1B:00.0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0,0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 [2026-10-16T13:13:23.921] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.925] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.925] debug2: ------------------------------- [2026-10-16T13:13:23.925] debug2: *9501 MHz [0] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *9251 MHz [1] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *5001 MHz [2] [2026-10-16T13:13:23.926] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.926] debug2: --------------------------------- [2026-10-16T13:13:23.926] debug2: *2100 MHz [0] [2026-10-16T13:13:23.926] debug2: *2085 MHz [1] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *1155 MHz [63] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *225 MHz [125] [2026-10-16T13:13:23.926] debug2: *210 MHz [126] [2026-10-16T13:13:23.926] debug2: *810 MHz [3] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *2100 MHz [0] [2026-10-16T13:13:23.927] debug2: *2085 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *1155 MHz [63] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [125] [2026-10-16T13:13:23.927] debug2: *210 MHz [126] [2026-10-16T13:13:23.927] debug2: *405 MHz [4] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *420 MHz [0] [2026-10-16T13:13:23.927] debug2: *405 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *315 MHz [7] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [13] [2026-10-16T13:13:23.927] debug2: *210 MHz [14] [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:30:0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1E:00.0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,0,0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia4 [2026-10-16T13:13:23.928] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.932] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.932] debug2: ------------------------------- [2026-10-16T13:13:23.932] debug2: *9501 MHz [0] [2026-10-16T13:13:23.932] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.932] debug2: --------------------------------- [2026-10-16T13:13:23.932] debug2: *2100 MHz [0] [2026-10-16T13:13:23.932] debug2: *2085 MHz [1] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *1155 MHz [63] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *225 MHz [125] [2026-10-16T13:13:23.932] debug2: *210 MHz [126] [2026-10-16T13:13:23.932] debug2: *9251 MHz [1] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *5001 MHz [2] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *810 MHz [3] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *2100 MHz [0] [2026-10-16T13:13:23.934] debug2: *2085 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *1155 MHz [63] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [125] [2026-10-16T13:13:23.934] debug2: *210 MHz [126] [2026-10-16T13:13:23.934] debug2: *405 MHz [4] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *420 MHz [0] [2026-10-16T13:13:23.934] debug2: *405 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *315 MHz [7] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [13] [2026-10-16T13:13:23.934] debug2: *210 MHz [14] [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100_80gb_pcie [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:63:0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:3F:00.0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,-1,0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 [2026-10-16T13:13:23.936] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.937] debug2: Possible GPU Memory Frequencies (1): [2026-10-16T13:13:23.937] debug2: ------------------------------- [2026-10-16T13:13:23.937] debug2: *1512 MHz [0] [2026-10-16T13:13:23.938] debug2: Possible GPU Graphics Frequencies (81): [2026-10-16T13:13:23.938] debug2: --------------------------------- [2026-10-16T13:13:23.938] debug2: *1410 MHz [0] [2026-10-16T13:13:23.938] debug2: *1395 MHz [1] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *810 MHz [40] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *225 MHz [79] [2026-10-16T13:13:23.938] debug2: *210 MHz [80] [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 4: [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:64:0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:40:00.0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,0,-1 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 [2026-10-16T13:13:23.938] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.942] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.942] debug2: ------------------------------- [2026-10-16T13:13:23.942] debug2: *9501 MHz [0] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *9251 MHz [1] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *5001 MHz [2] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *810 MHz [3] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *405 MHz [4] [2026-10-16T13:13:23.945] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.945] debug2: --------------------------------- [2026-10-16T13:13:23.945] debug2: *420 MHz [0] [2026-10-16T13:13:23.945] debug2: *405 MHz [1] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *315 MHz [7] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *225 MHz [13] [2026-10-16T13:13:23.945] debug2: *210 MHz [14] [2026-10-16T13:13:23.945] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:13:23.954] debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML [2026-10-16T13:13:23.954] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin [2026-10-16T13:13:23.955] Configured with rsmi, but rsmi isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] Configured with oneAPI, but oneAPI isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] debug: gpu/nrt: init: init: GPU NRT plugin loaded [2026-10-16T13:13:23.955] error: System GPU detection failed [2026-10-16T13:13:23.955] debug: gpu/nrt: fini: fini: unloading GPU NRT plugin NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-03:12:06 -Dj
The recommendation is in the error On 16/10/2026 20:20, Dj Merrill via slurm-users wrote:
Hi Reed! Thanks for your input!
The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz".
I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error:
From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration.
Did you try setting Parameters=l3cache_as_socket as per the error suggestion? You don't need to modify slurm.conf just yet. look at slurmd -C --parameters=l3cache_as_socket You can compare the result to nvidia-smi topo -m And if it aligns, modify slurm.conf to match the slurmd -C result. Let me show an example from an AMD node with 8xL40s GPUs Only showing the CPU Affinity column output of nvidia-smi topo -m:
nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-7]/ {printf "%-8s%-8s%s\n", $1, $14, $15}' CPU Affinity GPU0 16-31 1 GPU1 16-31 1 GPU2 0-15 0 GPU3 0-15 0 GPU4 80-95 5 GPU5 80-95 5 GPU6 64-79 4 GPU7 64-79 4
Note 4 affinity groups of 16 cores each. 0 0-15 GPU 2-3 1 16-31 GPU 0-1 4 64-79 GPU 6-7 5 80-95 GPU 4-5 Now the different slurmd -C results, to help visualize the comparison
slurmd -C NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=1547048 Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:48:48 64 cores per affinity group. This is the actual physical CPU layout, but it's wrong for my nodes. slurmd -C --parameters=l3cache_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=32 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=1547048 Parameters=l3cache_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:49:44 4 cores per affinity group - wrong for us slurmd -C --parameters=numa_node_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=8 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=1547048 Parameters=numa_node_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:50:01 16 cores per affinity group - this aligns with the nvidia-smi topo output
From your error I would guess that for your node you actually need l3cache_as_socket. The above is based on https://slurm.schedmd.com/gres.html#Socket_Affinity Another concern is your use of deprecated cgroupv01, but I don't think it's the cause of this issue. You should look into moving to cgroupv02, if your distro allows. Slurm has deprecated cgroupv01 and will remove support for it in future versions.
Hi Daniel! Thanks for chiming in! Adding "Parameters=l3cache_as_socket" to the node definition in slurm.conf was the first thing we tried based on the output of the error message. Trying your suggestion of the various options to slurmd -C shows no change: # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:34 # /opt/slurm/sbin/slurmd -C --parameters=l3cache_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=l3cache_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:52 # /opt/slurm/sbin/slurmd -C --parameters=numa_node_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=numa_node_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:01:19 # nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-4]/ {printf "%-8s%-8s%s\n", $1, $8, $9}' CPU Affinity GPU0 0-7 0 GPU1 0-7 0 GPU2 0-7 0 GPU3 0-7 0 GPU4 0-7 0 Beginning to think this might be a bug in Slurm version 25.11.6. I may try upgrading to 25.11.8. Re: your note about cgroups v1, we are in the early stages of building a new HPC Cluster built on Rocky Linux 10, and this will be using cgroups v2, and will be using the latest version of Slurm. -Dj On 10/17/26 05:04, Daniel Letai via slurm-users wrote:
The recommendation is in the error
On 16/10/2026 20:20, Dj Merrill via slurm-users wrote:
Hi Reed! Thanks for your input!
The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz".
I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error:
From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration.
Did you try setting Parameters=l3cache_as_socket as per the error suggestion?
You don't need to modify slurm.conf just yet.
look at
slurmd -C --parameters=l3cache_as_socket
You can compare the result to
nvidia-smi topo -m
And if it aligns, modify slurm.conf to match the slurmd -C result.
Let me show an example from an AMD node with 8xL40s GPUs
Only showing the CPU Affinity column output of nvidia-smi topo -m:
nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-7]/ {printf "%-8s%-8s%s\n", $1, $14, $15}' CPU Affinity GPU0 16-31 1 GPU1 16-31 1 GPU2 0-15 0 GPU3 0-15 0 GPU4 80-95 5 GPU5 80-95 5 GPU6 64-79 4 GPU7 64-79 4
Note 4 affinity groups of 16 cores each.
0 0-15 GPU 2-3
1 16-31 GPU 0-1
4 64-79 GPU 6-7
5 80-95 GPU 4-5
Now the different slurmd -C results, to help visualize the comparison
slurmd -C NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=1547048 Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:48:48 64 cores per affinity group. This is the actual physical CPU layout, but it's wrong for my nodes. slurmd -C --parameters=l3cache_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=32 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=1547048 Parameters=l3cache_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:49:44 4 cores per affinity group - wrong for us slurmd -C --parameters=numa_node_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=8 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=1547048 Parameters=numa_node_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:50:01 16 cores per affinity group - this aligns with the nvidia-smi topo output
From your error I would guess that for your node you actually need l3cache_as_socket.
The above is based on https://slurm.schedmd.com/gres.html#Socket_Affinity Another concern is your use of deprecated cgroupv01, but I don't think it's the cause of this issue. You should look into moving to cgroupv02, if your distro allows. Slurm has deprecated cgroupv01 and will remove support for it in future versions.
Hi Dj, We recently upgraded to v25.11.6 and experienced the same issue on our GPU nodes (AMD EPYC 7700 series CPUs + NVIDIA A100s). We observed significant differences in how `slurmd -G`enumerates NUMA nodes when setting `SlurmdParameter`to `l3cache_as_socket` and `numa_node_as_socket`. Comparing to the output of `lscpu`, we found numa_node_as_socket to match the node’s hardware configuration. The Slurm node config still didn’t match NVML output, which I believe is the cause of this error. We set `SlurmdParameter=numa_node_as_socket` and changed the `CoresPerSocket` and `SocketsPerBoard` in our GPU node definitions to match NVML’s output. This made Slurm happy. Let me know how it goes. Best, — Sebastian Smith Principal Systems Engineer | High-Performance Computing sebastian.smith@seattlechildrens.org Cell (360) 358-3283 SeattleChildrens.org [A black and grey logo Description automatically generated] From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> Date: Saturday, October 17, 2026 at 6:56 AM To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: Jobs not getting assigned the requested GPU card This Message Is From an External Sender Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/NuzbfyPwt6ZyPHQ!gfzioreFzBuCE7_cAXsNJ0f_Yn0VLa8KvOkdWE-90zf-QaE4IaB80wJpL9vSqTfpbPgHYgXf7d7UzKxvBid66Rguo2LPWCh3nF4P98ohL8s$> Hi Daniel! Thanks for chiming in! Adding "Parameters=l3cache_as_socket" to the node definition in slurm.conf was the first thing we tried based on the output of the error message. Trying your suggestion of the various options to slurmd -C shows no change: # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:34 # /opt/slurm/sbin/slurmd -C --parameters=l3cache_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=l3cache_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:52 # /opt/slurm/sbin/slurmd -C --parameters=numa_node_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=numa_node_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:01:19 # nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-4]/ {printf "%-8s%-8s%s\n", $1, $8, $9}' CPU Affinity GPU0 0-7 0 GPU1 0-7 0 GPU2 0-7 0 GPU3 0-7 0 GPU4 0-7 0 Beginning to think this might be a bug in Slurm version 25.11.6. I may try upgrading to 25.11.8. Re: your note about cgroups v1, we are in the early stages of building a new HPC Cluster built on Rocky Linux 10, and this will be using cgroups v2, and will be using the latest version of Slurm. -Dj On 10/17/26 05:04, Daniel Letai via slurm-users wrote:
The recommendation is in the error
On 16/10/2026 20:20, Dj Merrill via slurm-users wrote:
Hi Reed! Thanks for your input!
The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz".
I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error:
From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration.
Did you try setting Parameters=l3cache_as_socket as per the error suggestion?
You don't need to modify slurm.conf just yet.
look at
slurmd -C --parameters=l3cache_as_socket
You can compare the result to
nvidia-smi topo -m
And if it aligns, modify slurm.conf to match the slurmd -C result.
Let me show an example from an AMD node with 8xL40s GPUs
Only showing the CPU Affinity column output of nvidia-smi topo -m:
nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-7]/ {printf "%-8s%-8s%s\n", $1, $14, $15}' CPU Affinity GPU0 16-31 1 GPU1 16-31 1 GPU2 0-15 0 GPU3 0-15 0 GPU4 80-95 5 GPU5 80-95 5 GPU6 64-79 4 GPU7 64-79 4
Note 4 affinity groups of 16 cores each.
0 0-15 GPU 2-3
1 16-31 GPU 0-1
4 64-79 GPU 6-7
5 80-95 GPU 4-5
Now the different slurmd -C results, to help visualize the comparison
slurmd -C NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=1547048 Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:48:48 64 cores per affinity group. This is the actual physical CPU layout, but it's wrong for my nodes. slurmd -C --parameters=l3cache_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=32 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=1547048 Parameters=l3cache_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:49:44 4 cores per affinity group - wrong for us slurmd -C --parameters=numa_node_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=8 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=1547048 Parameters=numa_node_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:50:01 16 cores per affinity group - this aligns with the nvidia-smi topo output
From your error I would guess that for your node you actually need l3cache_as_socket.
The above is based on https://urldefense.com/v3/__https://slurm.schedmd.com/gres.html*Socket_Affinity__;Iw!!NuzbfyPwt6ZyPHQ!vKa6FeMDaSyPR_pWRCjluUnBve_e6rrnEdoGNqumW-966cYLVI3t3xg8-OdUqlEuNz4NV36aivr_wY4ElLaNBfhN7JRtU4jNFfz6uWk$[slurm[.]schedmd[.]com] Another concern is your use of deprecated cgroupv01, but I don't think it's the cause of this issue. You should look into moving to cgroupv02, if your distro allows. Slurm has deprecated cgroupv01 and will remove support for it in future versions.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
The output you shared from 'nvidai-smi topo -m' and the 'slurmd -G' really seems to point that the GPUs are associated to the first socket with cores 0-7. Could you share the output of 'lscpu' and 'dmidecode -t system' (I'm just after the brand & model info of the server). I'd like to see what lscpu shows for the numa domains related to the l3cache as socket angle. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Friday, October 16, 2026 9:03 AM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Thanks, Brian. Trying the NVML autodetect approach again today. sinfo is showing the node status as "invalid" after changing to autodetect. slurmctld.log is complaining about socket boundaries: [2026-10-16T10:52:08.390] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T10:52:08.390] error: _slurm_rpc_node_registration node=moose66: Invalid argument Adding "Parameters=l3cache_as_socket" to the node config does not correct this issue. Node /etc/slurm/gres.conf: AutoDetect=nvml Node entry in slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=l3cache_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 # /opt/slurm/sbin/slurmd -G [2026-10-16T10:54:40.584] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T10:54:40.687] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:53:39 -Dj On 10/15/26 20:57, Brian Haymore wrote: The -C is the more important one for what I'm asking I think. It is showing that it was able to detect the GPUs with nvml. Wanted to make sure your slurm build & env had what was needed. When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:53 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card It appears that slurmd -G aligns with the devices shown by nvidia-smi -L: # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # /opt/slurm/sbin/slurmd -G [2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT [2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-00:06:34 -Dj On 10/15/26 20:45, Brian Haymore wrote: Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:40 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either. -Dj On 10/15/26 20:37, Brian Haymore wrote:
From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee. If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well. Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"
Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it. If your able to use the NVML autodetect method for the gres.conf that also addresses this. This is the direction we went after realizing this point. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net><mailto:slurm@deej.net> Sent: Thursday, October 15, 2026 6:02 PM To: Brian Haymore <brian.haymore@utah.edu><mailto:brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thank you for the input! I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either. As you surmised, the minor does not match the ID: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 4 ID: 1 Minor Number : 3 ID: 2 Minor Number : 2 ID: 3 Minor Number : 0 ID: 4 Minor Number : 1 Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same: # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 1 ID: 1 Minor Number : 2 ID: 2 Minor Number : 3 ID: 3 Minor Number : 0 ID: 4 Minor Number : 4 # for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done ID: 0 Minor Number : 2 ID: 1 Minor Number : 4 ID: 2 Minor Number : 3 ID: 3 Minor Number : 1 ID: 4 Minor Number : 0 # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3) # nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PIX PIX NODE NODE NODE 0-7 0 N/A GPU1 PIX X PIX NODE NODE NODE 0-7 0 N/A GPU2 PIX PIX X NODE NODE NODE 0-7 0 N/A GPU3 NODE NODE NODE X PIX PIX 0-7 0 N/A GPU4 NODE NODE NODE PIX X PIX 0-7 0 N/A NIC0 NODE NODE NODE PIX PIX X I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago. -Dj On 10/15/26 19:14, Brian Haymore wrote: The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file. Another command that would be good to see is 'nvidia-smi topo -m'. -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Sent: Thursday, October 15, 2026 1:42 PM To: slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> <slurm-users@lists.schedmd.com><mailto:slurm-users@lists.schedmd.com> Subject: [slurm-users] Jobs not getting assigned the requested GPU card We have a Slurm Cluster that has been working fine for many years. A few months ago it was upgraded to Slurm version 25.11.65, and has been working properly running that version. The systems are running Rocky Linux 9.8 and are patched/updated on a regular basis. They are currently running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from Nvidia's repo. Today it was reported that jobs are not getting assigned the GPU card they requested, so for example they may request a 5090 card but are assigned an A100 card. This was working fine, so we are struggling to figure out what has changed to cause this behaviour. We have a node with 5 GPU cards, of three types: % nvidia-smi -L GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6) GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94) GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae) GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094) /etc/slurm/gres.conf matches the above output (and has been this way for years): % cat /etc/slurm/gres.conf Name=gpu Type=rtx5090 File=/dev/nvidia0 Name=gpu Type=rtx3080 File=/dev/nvidia1 Name=gpu Type=rtx3080 File=/dev/nvidia2 Name=gpu Type=a100 File=/dev/nvidia3 Name=gpu Type=rtx3080 File=/dev/nvidia4 The /etc/slurm/slurm.conf entry for this node is the same as it has been for years: NodeName=moose[63] CPUs=16 RealMemory=380000 Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700 When submitting a job such as: $ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash srun: job 71048 queued and waiting for resources srun: job 71048 has been allocated resources $ nvidia-smi -L GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-c6be57f5-376d-2d06-d58b-20924b4296e7) Any suggestions as to where to look? Thank you, -Dj -- slurm-users mailing list -- slurm-users@lists.schedmd.com<mailto:slurm-users@lists.schedmd.com> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com<mailto:slurm-users-leave@lists.schedmd.com>
Hi Brian, Thanks for continuing to help with this. # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz BIOS Model name: Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz CPU family: 6 Model: 85 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 2 Stepping: 7 CPU(s) scaling MHz: 94% CPU max MHz: 3700.0000 CPU min MHz: 1200.0000 BogoMIPS: 6000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pg e mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm cons tant_tsc art arch_perfmon pebs bts rep_good nopl xt opology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx 16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe po pcnt tsc_deadline_timer aes xsave avx f16c rdrand l ahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp _l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanc ed fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms inv pcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512v l xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_ llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_ capabilities Caches (sum of all): L1d: 512 KiB (16 instances) L1i: 512 KiB (16 instances) L2: 16 MiB (16 instances) L3: 22 MiB (2 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 Vulnerabilities: Gather data sampling: Mitigation; Microcode Indirect target selection: Mitigation; Aligned branch/return thunks Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Mitigation; Clear CPU buffers; SMT disabled Old microcode: Not affected Reg file data sampling: Not affected Retbleed: Mitigation; Enhanced IBRS Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via p rctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user poi nter sanitization Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditi onal; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Srbds: Not affected Tsa: Not affected Tsx async abort: Mitigation; TSX disabled Vmscape: Mitigation; IBPB before exit to userspace # dmidecode -t system # dmidecode 3.6 Getting SMBIOS data from sysfs. SMBIOS 3.2.1 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Supermicro Product Name: SYS-4029GP-TRT2 Version: 123456789 Wake-up Type: Power Switch SKU Number: 1A0615D9 Family: SMC X11 -Dj On 10/19/26 13:11, Brian Haymore wrote:
The output you shared from 'nvidai-smi topo -m' and the 'slurmd -G' really seems to point that the GPUs are associated to the first socket with cores 0-7. Could you share the output of 'lscpu' and 'dmidecode -t system' (I'm just after the brand & model info of the server). I'd like to see what lscpu shows for the numa domains related to the l3cache as socket angle.
Ok from the lscpu output you are just a single numa domain per socket. From the nvidia-smi topo -m it seems to paint that all of the GPUs are on the first socket. Does that match what you expect on this server? -- Brian D. Haymore University of Utah Center for High Performance Computing 155 South 1452 East RM 405 Salt Lake City, Ut 84112 Phone: 801-558-1150 http://bit.ly/1HO1N2C ________________________________ From: Dj Merrill <slurm@deej.net> Sent: Monday, October 19, 2026 11:20 AM To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com> Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card Hi Brian, Thanks for continuing to help with this. # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: GenuineIntel BIOS Vendor ID: Intel(R) Corporation Model name: Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz BIOS Model name: Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz CPU family: 6 Model: 85 Thread(s) per core: 1 Core(s) per socket: 8 Socket(s): 2 Stepping: 7 CPU(s) scaling MHz: 94% CPU max MHz: 3700.0000 CPU min MHz: 1200.0000 BogoMIPS: 6000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pg e mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm cons tant_tsc art arch_perfmon pebs bts rep_good nopl xt opology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx 16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe po pcnt tsc_deadline_timer aes xsave avx f16c rdrand l ahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp _l3 intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanc ed fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms inv pcid cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512v l xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_ llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_ capabilities Caches (sum of all): L1d: 512 KiB (16 instances) L1i: 512 KiB (16 instances) L2: 16 MiB (16 instances) L3: 22 MiB (2 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-7 NUMA node1 CPU(s): 8-15 Vulnerabilities: Gather data sampling: Mitigation; Microcode Indirect target selection: Mitigation; Aligned branch/return thunks Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Mitigation; Clear CPU buffers; SMT disabled Old microcode: Not affected Reg file data sampling: Not affected Retbleed: Mitigation; Enhanced IBRS Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via p rctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user poi nter sanitization Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditi onal; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Srbds: Not affected Tsa: Not affected Tsx async abort: Mitigation; TSX disabled Vmscape: Mitigation; IBPB before exit to userspace # dmidecode -t system # dmidecode 3.6 Getting SMBIOS data from sysfs. SMBIOS 3.2.1 present. Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Supermicro Product Name: SYS-4029GP-TRT2 Version: 123456789 Wake-up Type: Power Switch SKU Number: 1A0615D9 Family: SMC X11 -Dj On 10/19/26 13:11, Brian Haymore wrote: The output you shared from 'nvidai-smi topo -m' and the 'slurmd -G' really seems to point that the GPUs are associated to the first socket with cores 0-7. Could you share the output of 'lscpu' and 'dmidecode -t system' (I'm just after the brand & model info of the server). I'd like to see what lscpu shows for the numa domains related to the l3cache as socket angle.
Thank you to everyone that offered help on this. I really appreciate it! I did figure out a solution, and it was stupid simple. This is the original node definition in slurm.conf that gave the "doesn't match socket boundaries" error in the logs when "AutoDetect=nvml" was enabled on the node: NodeName=moose[66] CPUs=16 RealMemory=380000 Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 This works: NodeName=moose[66] CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=380000 Gres=gpu:rtx_5090:1,gpu:rtx_3080:3,gpu:a100:1 Weight=700 Basically all I had to do was add the other parameters that slurmd -C printed out to the node definition. Autodetect is working properly, and we are now getting the proper requested GPU card assigned to jobs. -Dj
participants (5)
-
Brian Haymore -
Daniel Letai -
Dj Merrill -
Reed Dier -
Smith, Sebastian