I'll hazard a guess that the cpu on moose66 is an AMD?
I've seen this issue on a Naples and Rome systems with GPUs.

I had to use this to get it to play nice on the AMD boxes (due to NUMA topology of the AMD chiplet design, verifiable with `numactl --hardware`, and I'm using AutoDetect=nvml
SlurmdParameters=numa_node_as_socket

I do not profess to be an expert on this, but I do know that the _merge_system_gres_conf funciton is the one you're fighting with as it pertains to the naming of the gres.

If you throw a few verbose flags at slurmd you can get some good debug output detailing what the nvml autodetect is seeing and doing.

slurmd -C -vvv
slurmd -G -vv

I've never had issues with named substrings
Found gpu:tesla_p40:6,gpu:quadro_rtx_6000:2 with Autodetect=nvml (Substring of gpu name may be used instead)

[2026-10-16T12:33:03.711] Gres GPU plugin: Final merged GRES list:
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:-1,0,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-9c8c4abb-4099-e2e5-11c6-f290d767a767
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:0,-1,0,0,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-72e391a7-be28-b300-8a4b-ea8469531950
[2026-10-16T12:33:03.711]     GRES[gpu] Type:rtx Count:1 Cores(88):0-21  Links:0,0,-1,2,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-e0ef1ca0-f749-8e4f-b9f7-a36d90f8b27e
[2026-10-16T12:33:03.711]     GRES[gpu] Type:rtx Count:1 Cores(88):0-21  Links:0,0,2,-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-fc568a4e-c008-e414-63af-52d29d4ba511
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:0,0,0,0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-b4c05798-be9e-b1d7-e1ff-d6f9a8e64ddf
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:0,0,0,0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia5 UniqueId:GPU-1eddadc4-edb3-99ba-16d6-0382df255265
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:0,0,0,0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia6 UniqueId:GPU-e7f4838f-5986-4d13-6aa3-4e54bdec082a
[2026-10-16T12:33:03.711]     GRES[gpu] Type:p40 Count:1 Cores(88):0-21  Links:0,0,0,0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia7 UniqueId:GPU-f002a9a2-119f-a54e-f6ff-db9a15d411d7
[2026-10-16T12:33:03.711]     GRES[shard] Type:(null) Count:48 Cores(88):(null)  Links:(null) Flags: File:(null) UniqueId:(null)

For instance p40 and rtx (defined in slurm.conf) are both substrings of tesla_p40 and quadro_rtx_6000 respectively.

I'm not sure if `rtx5090` would "match" `nvidia_geforce_rtx_5090`.
I suspect `rtx_5090` would, if `rtx5090` did not, and I feel more sure that `5090` would match.

If I had some idle systems to test against, I'd love to prove or disprove this theory.

Hope this is at least somewhat helpful,
Reed

On Oct 16, 2026, at 10:03 AM, Dj Merrill via slurm-users <slurm-users@lists.schedmd.com> wrote:

Thanks, Brian.   Trying the NVML autodetect approach again today.

sinfo is showing the node status as "invalid" after changing to autodetect.

slurmctld.log is complaining about socket boundaries:

[2026-10-16T10:52:08.390] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration.
[2026-10-16T10:52:08.390] error: _slurm_rpc_node_registration node=moose66: Invalid argument


Adding "Parameters=l3cache_as_socket" to the node config does not correct this issue.


Node /etc/slurm/gres.conf:
AutoDetect=nvml

Node entry in slurm.conf:
NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=l3cache_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700


# /opt/slurm/sbin/slurmd -G
[2026-10-16T10:54:40.584] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience
[2026-10-16T10:54:40.687] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected
[2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
[2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
[2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
[2026-10-16T10:54:40.687] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML
[2026-10-16T10:54:40.687] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML


# /opt/slurm/sbin/slurmd -C
NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead)
UpTime=0-00:53:39


-Dj



On 10/15/26 20:57, Brian Haymore wrote:
The -C is the more important one for what I'm asking I think.  It is showing that it was able to detect the GPUs with nvml.  Wanted to make sure your slurm build & env had what was needed.

When you tried to use autodetect, how did you craft the lines in the gres.conf for the node and what did you have in the nodename line in your slurm.conf for the node?

--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150
http://bit.ly/1HO1N2C
From: Dj Merrill <slurm@deej.net>
Sent: Thursday, October 15, 2026 6:53 PM
To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
 
It appears that slurmd -G aligns with the devices shown by nvidia-smi -L:

# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd)
GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595)
GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850)
GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)

# /opt/slurm/sbin/slurmd -G
[2026-10-15T20:49:20.849] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience
[2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2026-10-15T20:49:20.856] Gres Name=gpu Type=rtx5090 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT
[2026-10-15T20:49:20.856] Gres Name=gpu Type=a100 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Links=(null) Flags=HAS_FILE,HAS_TYPE,ENV_NVML,ENV_RSMI,ENV_ONEAPI,ENV_OPENCL,ENV_DEFAULT

# /opt/slurm/sbin/slurmd -C
NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead)
UpTime=0-00:06:34

-Dj


On 10/15/26 20:45, Brian Haymore wrote:
Can you look at what '<your path to slurmd>/slurmd -G' and '<your path to slurmd>/slurmd -C' show?  

--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150
http://bit.ly/1HO1N2C
From: Dj Merrill <slurm@deej.net>
Sent: Thursday, October 15, 2026 6:40 PM
To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
 
We tried both "AutoDetect=nvml" and "AutoDetect=nvidia" in the gres.conf and neither of those worked properly either.

-Dj


On 10/15/26 20:37, Brian Haymore wrote:
>From my understanding nvidia-smi enumerates the devices by the PCI bus ID which on some systems could match the order the devices as initialized by the driver but that this is not a guarantee.  If you look at the 'man nvidia-smi' on the '-i' param section it touches on this and over on https://slurm.schedmd.com/gres.html it notes this as well.  Search down for where the paragraph starts with, "GPU device files (e.g. /dev/nvidia1) are based on the Linux minor number assignment"

Like you observed, this doesn't seem to be a regular event to have them change unless I make a hardware change where I should expect it.  If your able to use the NVML autodetect method for the gres.conf that also addresses this.  This is the direction we went after realizing this point.

--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150
http://bit.ly/1HO1N2C
From: Dj Merrill <slurm@deej.net>
Sent: Thursday, October 15, 2026 6:02 PM
To: Brian Haymore <brian.haymore@utah.edu>; slurm-users <slurm-users@lists.schedmd.com>
Subject: Re: [slurm-users] Jobs not getting assigned the requested GPU card
 
Hi Brian,
Thank you for the input!
I find this weird and interesting.  For the past several years we have gone by the output of nvidia-smi -L to assign the GPUs in the gres.conf, and that's always worked. 

We also tried removing the fixed settings and tried both AutoDetect=nvml and AutoDetect=nvidia in the gres.conf and neither of those worked properly either.

As you surmised, the minor does not match the ID:

# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 4
ID: 1
    Minor Number                                       : 3
ID: 2
    Minor Number                                       : 2
ID: 3
    Minor Number                                       : 0
ID: 4
    Minor Number                                       : 1


Interestingly, every time I reboot, I get a different result for the minor number, but the output of "nvidia-smi -L" always stays the same:


# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 1
ID: 1
    Minor Number                                       : 2
ID: 2
    Minor Number                                       : 3
ID: 3
    Minor Number                                       : 0
ID: 4
    Minor Number                                       : 4


# for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"; done
ID: 0
    Minor Number                                       : 2
ID: 1
    Minor Number                                       : 4
ID: 2
    Minor Number                                       : 3
ID: 3
    Minor Number                                       : 1
ID: 4
    Minor Number                                       : 0


# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd)
GPU 2: NVIDIA GeForce RTX 3080 (UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595)
GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-418da940-1928-71e6-fa74-8889e551d850)
GPU 4: NVIDIA GeForce RTX 3080 (UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3)


# nvidia-smi topo -m
       GPU0    GPU1    GPU2    GPU3    GPU4    NIC0    CPU Affinity    NUMA Affinity    GPU NUMA ID
GPU0     X     PIX    PIX    NODE    NODE    NODE    0-7    0        N/A
GPU1    PIX     X     PIX    NODE    NODE    NODE    0-7    0        N/A
GPU2    PIX    PIX     X     NODE    NODE    NODE    0-7    0        N/A
GPU3    NODE    NODE    NODE     X     PIX    PIX    0-7    0        N/A
GPU4    NODE    NODE    NODE    PIX     X     PIX    0-7    0        N/A
NIC0    NODE    NODE    NODE    PIX    PIX     X                 


I'm wondering if something changed in the recent nvidia driver / CUDA that was installed a few days ago.


-Dj



On 10/15/26 19:14, Brian Haymore wrote:
The device number that 'nvidia-smi' shows for the GPUs, 0, 1, 2, 3, and 4, are not certain to map to /dev/nvidia$X. 

Could you run ' for i in 0 1 2 3 4; do echo "ID: $i"; nvidia-smi -q -i $i | grep "Minor Number"'; done

If each matches then what I'm asking about is not the issue.  If they do not match then you will need to adjust the gres.conf file.

Another command that would be good to see is 'nvidia-smi topo -m'.

--
Brian D. Haymore
University of Utah
Center for High Performance Computing
155 South 1452 East RM 405
Salt Lake City, Ut 84112
Phone: 801-558-1150
http://bit.ly/1HO1N2C
From: Dj Merrill via slurm-users <slurm-users@lists.schedmd.com>
Sent: Thursday, October 15, 2026 1:42 PM
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Jobs not getting assigned the requested GPU card
 
We have a Slurm Cluster that has been working fine for many years.  A
few months ago it was upgraded to Slurm version 25.11.65, and has been 
working properly running that version.  The systems are running Rocky 
Linux 9.8 and are patched/updated on a regular basis. They are currently 
running CUDA 13.4 with Nvidia driver 615.71.09 installed via dnf from 
Nvidia's repo.

Today it was reported that jobs are not getting assigned the GPU card 
they requested, so for example they may request a 5090 card but are 
assigned an A100 card.  This was working fine, so we are struggling to 
figure out what has changed to cause this behaviour.

We have a node with 5 GPU cards, of three types:

% nvidia-smi -L

GPU 0: NVIDIA GeForce RTX 5090 (UUID: 
GPU-8efff06f-af9c-4598-420f-fcbb5b8af9f6)
GPU 1: NVIDIA GeForce RTX 3080 (UUID: 
GPU-1d473ffd-d18c-e1f6-c492-02f4d702fd94)
GPU 2: NVIDIA GeForce RTX 3080 (UUID: 
GPU-aed53cd7-6a97-c62d-e825-bc5be78be5ae)
GPU 3: NVIDIA A100 80GB PCIe (UUID: 
GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)
GPU 4: NVIDIA GeForce RTX 3080 (UUID: 
GPU-3f7e5512-c25b-6dac-cbfa-2a7b95267094)

/etc/slurm/gres.conf matches the above output (and has been this way for 
years):

% cat /etc/slurm/gres.conf
Name=gpu Type=rtx5090  File=/dev/nvidia0
Name=gpu Type=rtx3080  File=/dev/nvidia1
Name=gpu Type=rtx3080  File=/dev/nvidia2
Name=gpu Type=a100  File=/dev/nvidia3
Name=gpu Type=rtx3080  File=/dev/nvidia4


The /etc/slurm/slurm.conf entry for this node is the same as it has been 
for years:

NodeName=moose[63] CPUs=16 RealMemory=380000 
Gres=gpu:rtx3080:3,gpu:rtx5090:1,gpu:a100:1 Weight=700


When submitting a job such as:

$ srun -p gpu --gres=gpu:rtx5090:1 --pty /bin/bash
srun: job 71048 queued and waiting for resources
srun: job 71048 has been allocated resources

$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: 
GPU-c6be57f5-376d-2d06-d58b-20924b4296e7)

Any suggestions as to where to look?

Thank you,

-Dj




-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com





-- 
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com