This Message Is From an External Sender
Hi Daniel!
Thanks for chiming in!
Adding "Parameters=l3cache_as_socket" to the node definition in
slurm.conf was the first thing we tried based on the output of the error
message.
Trying your suggestion of the various options to slurmd -C shows no change:
# /opt/slurm/sbin/slurmd -C
NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8
ThreadsPerCore=1 RealMemory=385100
Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
Found
gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
with Autodetect=nvml (Substring of gpu name may be used instead)
UpTime=0-23:00:34
# /opt/slurm/sbin/slurmd -C --parameters=l3cache_as_socket
NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8
ThreadsPerCore=1 RealMemory=385100 Parameters=l3cache_as_socket
Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
Found
gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
with Autodetect=nvml (Substring of gpu name may be used instead)
UpTime=0-23:00:52
# /opt/slurm/sbin/slurmd -C --parameters=numa_node_as_socket
NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8
ThreadsPerCore=1 RealMemory=385100 Parameters=numa_node_as_socket
Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
Found
gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1
with Autodetect=nvml (Substring of gpu name may be used instead)
UpTime=0-23:01:19
# nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU Affinity"}
/^GPU[0-4]/ {printf "%-8s%-8s%s\n", $1, $8, $9}'
CPU Affinity
GPU0 0-7 0
GPU1 0-7 0
GPU2 0-7 0
GPU3 0-7 0
GPU4 0-7 0
Beginning to think this might be a bug in Slurm version 25.11.6. I may
try upgrading to 25.11.8.
Re: your note about cgroups v1, we are in the early stages of building a
new HPC Cluster built on Rocky Linux 10, and this will be using cgroups
v2, and will be using the latest version of Slurm.
-Dj
On 10/17/26 05:04, Daniel Letai via slurm-users wrote:
> The recommendation is in the error
>
> On 16/10/2026 20:20, Dj Merrill via slurm-users wrote:
>> Hi Reed!
>> Thanks for your input!
>>
>> The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz".
>>
>> I tried changing the node definition in slurm.conf to
>> "Parameters=numa_node_as_socket", but I still get the same error:
>>
>> From slurm.conf:
>> NodeName=moose[66] CPUs=16 RealMemory=380000
>> Parameters=numa_node_as_socket
>> Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700
>>
>>
>> From slurmctld.log:
>> [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock:
>> gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't
>> match socket boundaries. (Socket 0 is cores 0-0). Consider setting
>> Parameters=l3cache_as_socket as part of the Node configuration.
>
> Did you try setting Parameters=l3cache_as_socket as per the error
> suggestion?
>
> You don't need to modify slurm.conf just yet.
>
> look at
>
> slurmd -C --parameters=l3cache_as_socket
>
> You can compare the result to
>
> nvidia-smi topo -m
>
> And if it aligns, modify slurm.conf to match the slurmd -C result.
>
> Let me show an example from an AMD node with 8xL40s GPUs
>
> Only showing the CPU Affinity column output of nvidia-smi topo -m:
>
>> nvidia-smi topo -m | awk 'BEGIN {printf " %s\n", "CPU
>> Affinity"} /^GPU[0-7]/ {printf "%-8s%-8s%s\n", $1, $14, $15}'
>> CPU Affinity
>> GPU0 16-31 1
>> GPU1 16-31 1
>> GPU2 0-15 0
>> GPU3 0-15 0
>> GPU4 80-95 5
>> GPU5 80-95 5
>> GPU6 64-79 4
>> GPU7 64-79 4
>
> Note 4 affinity groups of 16 cores each.
>
> 0 0-15 GPU 2-3
>
> 1 16-31 GPU 0-1
>
> 4 64-79 GPU 6-7
>
> 5 80-95 GPU 4-5
>
> Now the different slurmd -C results, to help visualize the comparison
>
>
>> slurmd -C
>> NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=2 CoresPerSocket=64
>> ThreadsPerCore=1 RealMemory=1547048 Gres=gpu:nvidia_l40s:8
>> Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name
>> may be used instead)
>> UpTime=5-16:48:48
> 64 cores per affinity group. This is the actual physical CPU layout,
> but it's wrong for my nodes.
>> slurmd -C --parameters=l3cache_as_socket
>> NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=32 CoresPerSocket=4
>> ThreadsPerCore=1 RealMemory=1547048 Parameters=l3cache_as_socket
>> Gres=gpu:nvidia_l40s:8
>> Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name
>> may be used instead)
>> UpTime=5-16:49:44
> 4 cores per affinity group - wrong for us
>> slurmd -C --parameters=numa_node_as_socket
>> NodeName=r13g01 CPUs=128 Boards=1 SocketsPerBoard=8 CoresPerSocket=16
>> ThreadsPerCore=1 RealMemory=1547048 Parameters=numa_node_as_socket
>> Gres=gpu:nvidia_l40s:8
>> Found gpu:nvidia_l40s:8 with Autodetect=nvml (Substring of gpu name
>> may be used instead)
>> UpTime=5-16:50:01
> 16 cores per affinity group - this aligns with the nvidia-smi topo output
>
> From your error I would guess that for your node you actually
> need l3cache_as_socket.
>
> The above is based on
https://urldefense.com/v3/__https://slurm.schedmd.com/gres.html*Socket_Affinity__;Iw!!NuzbfyPwt6ZyPHQ!vKa6FeMDaSyPR_pWRCjluUnBve_e6rrnEdoGNqumW-966cYLVI3t3xg8-OdUqlEuNz4NV36aivr_wY4ElLaNBfhN7JRtU4jNFfz6uWk$[slurm[.]schedmd[.]com]
> Another concern is your use of deprecated cgroupv01, but I don't think
> it's the cause of this issue. You should look into moving to
> cgroupv02, if your distro allows. Slurm has deprecated cgroupv01 and
> will remove support for it in future versions.
>
--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com