Hi Brian,

Thank you for the input!

I find this weird and interesting. For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked.

We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.

As you surmised, the minor does not match the ID:

# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 4
ID: 1
    Minor Number                                       : 3
ID: 2
    Minor Number                                       : 2
ID: 3
    Minor Number                                       : 0
ID: 4
    Minor Number                                       : 1

Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:

# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 1
ID: 1
    Minor Number                                       : 2
ID: 2
    Minor Number                                       : 3
ID: 3
    Minor Number                                       : 0
ID: 4
    Minor Number                                       : 4

# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 2
ID: 1
    Minor Number                                       : 4
ID: 2
    Minor Number                                       : 3
ID: 3
    Minor Number                                       : 1
ID: 4
    Minor Number                                       : 0

# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd)
GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595)
GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850)
GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)

# nvidia-smi topo -m
       GPU0    GPU1    GPU2    GPU3    GPU4    NIC0    CPU Affinity    NUMA Affinity    GPU NUMA ID
GPU0    X     PIX    PIX    NODE    NODE    NODE    0-7    0        N/A
GPU1    PIX    X     PIX    NODE    NODE    NODE    0-7    0        N/A
GPU2    PIX    PIX    X     NODE    NODE    NODE    0-7    0        N/A
GPU3    NODE    NODE    NODE    X     PIX    PIX    0-7    0        N/A
GPU4    NODE    NODE    NODE    PIX    X     PIX    0-7    0        N/A
NIC0    NODE    NODE    NODE    PIX    PIX    X

I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.

-Dj

On 10/15/26 19:14, Brian Haymore wrote:

The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X.

Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done

If each matches then what I'm asking about is not the issue. If they do not match then you will need to adjust the gres.conf file.

Another command that would be good to see is 'nvidia-smi topo -m'.

--

Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150
http://bit.ly/1HO1N2C

From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com>
Sent: Thursday, October 15, 2026 1:42 PM
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Jobs not getting assigned the requested GPU card

We have a Slurm Cluster that has been working fine for many years. A
few months ago it was upgraded to Slurm version 25.11.65, and has been
working properly running that version. The systems are running Rocky
Linux 9.8 and are patched/updated on a regular basis. They are currently
running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from
Nvidia's repo.

Today it was reported that jobs are not getting assigned the GPU card
they requested, so for example they may request a 5090 card but are
assigned an A100 card. This was working fine, so we are struggling to
figure out what has changed to cause this behaviour.

We have a node with 5 GPU cards, of three types:

% nvidia-smi -L

GPU 0: NVIDIA GeForce RTX 5090 (UUID:
GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6)
GPU 1: NVIDIA GeForce RTX 3080 (UUID:
GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94)
GPU 2: NVIDIA GeForce RTX 3080 (UUID:
GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae)
GPU 3: NVIDIA A100 80GB PCIe (UUID:
GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
GPU 4: NVIDIA GeForce RTX 3080 (UUID:
GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)

/etc/slurm/gres.conf matches the above output (and has been this way for
years):

% cat /etc/slurm/gres.conf
Name=gpu Type=rtx5090 File=/dev/nvidia0
Name=gpu Type=rtx3080 File=/dev/nvidia1
Name=gpu Type=rtx3080 File=/dev/nvidia2
Name=gpu Type=a100 File=/dev/nvidia3
Name=gpu Type=rtx3080 File=/dev/nvidia4

The /etc/slurm/slurm.conf entry for this node is the same as it has been
for years:

NodeName=moose[63] CPUs=16 RealMemory=380000
Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700

When submitting a job such as:

$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash
srun: job 71048 queued and waiting for resources
srun: job 71048 has been allocated resources

$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID:
GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)

Any suggestions as to where to look?

Thank you,

-Dj

--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com