Hi Daniel! Thanks for chiming in! Adding "Parameters=l3cache_as_socket" to the node definition in slurm.conf was the first thing we tried based on the output of the error message. Trying your suggestion of the various options to slurmd -C shows no change: # /opt/slurm/sbin/slurmd -C NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:34 # /opt/slurm/sbin/slurmd -C --parameters=l3cache_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=l3cache_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:00:52 # /opt/slurm/sbin/slurmd -C --parameters=numa_node_as_socket NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Parameters=numa_node_as_socket Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-23:01:19 # nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-4]/ {printf "%-8s%-8s%s\n", $1, $8, $9}' CPU Affinity GPU0 0-7 0 GPU1 0-7 0 GPU2 0-7 0 GPU3 0-7 0 GPU4 0-7 0 Beginning to think this might be a bug in Slurm version 25.11.6. I may try upgrading to 25.11.8. Re: your note about cgroups v1, we are in the early stages of building a new HPC Cluster built on Rocky Linux 10, and this will be using cgroups v2, and will be using the latest version of Slurm. -Dj On 10/17/26 05:04, Daniel Letai via slurm-users wrote:
The recommendation is in the error
On 16/10/2026 20:20, Dj Merrill via slurm-users wrote:
Hi Reed! Thanks for your input!
The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz".
I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error:
From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration.
Did you try setting Parameters=l3cache_as_socket as per the error suggestion?
You don't need to modify slurm.conf just yet.
look at
slurmd -C --parameters=l3cache_as_socket
You can compare the result to
nvidia-smi topo -m
And if it aligns, modify slurm.conf to match the slurmd -C result.
Let me show an example from an AMD node with 8xL40s GPUs
Only showing the CPU Affinity column output of nvidia-smi topo -m:
nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"} /^GPU[0-7]/ {printf "%-8s%-8s%s\n", $1, $14, $15}' CPU Affinity GPU0 16-31 1 GPU1 16-31 1 GPU2 0-15 0 GPU3 0-15 0 GPU4 80-95 5 GPU5 80-95 5 GPU6 64-79 4 GPU7 64-79 4
Note 4 affinity groups of 16 cores each.
0 0-15 GPU 2-3
1 16-31 GPU 0-1
4 64-79 GPU 6-7
5 80-95 GPU 4-5
Now the different slurmd -C results, to help visualize the comparison
slurmd -C NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1 RealMemory=1547048 Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:48:48 64 cores per affinity group. This is the actual physical CPU layout, but it's wrong for my nodes. slurmd -C --parameters=l3cache_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=32 CoresPerSocket=4 ThreadsPerCore=1 RealMemory=1547048 Parameters=l3cache_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:49:44 4 cores per affinity group - wrong for us slurmd -C --parameters=numa_node_as_socket NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=8 CoresPerSocket=16 ThreadsPerCore=1 RealMemory=1547048 Parameters=numa_node_as_socket Gres=gpu:nvidia_l40s:8 Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=5-16:50:01 16 cores per affinity group - this aligns with the nvidia-smi topo output
From your error I would guess that for your node you actually need l3cache_as_socket.
The above is based on https://slurm.schedmd.com/gres.html#Socket_Affinity Another concern is your use of deprecated cgroupv01, but I don't think it's the cause of this issue. You should look into moving to cgroupv02, if your distro allows. Slurm has deprecated cgroupv01 and will remove support for it in future versions.