Hi Reed! Thanks for your input! The CPU is an Intel chip, "Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz". I tried changing the node definition in slurm.conf to "Parameters=numa_node_as_socket", but I still get the same error: From slurm.conf: NodeName=moose[66] CPUs=16 RealMemory=380000 Parameters=numa_node_as_socket Gres=gpu:rtx_3080:3,gpu:rtx_5090:1,gpu:a100:1 Weight=700 From slurmctld.log: [2026-10-16T13:06:29.573] error: _check_core_range_matches_sock: gres/gpu GRES autodetected core affinity 0-7 on node moose66 doesn't match socket boundaries. (Socket 0 is cores 0-0). Consider setting Parameters=l3cache_as_socket as part of the Node configuration. [2026-10-16T13:06:29.573] error: _slurm_rpc_node_registration node=moose66: Invalid argument Here is the more verbose output you asked for: # /opt/slurm/sbin/slurmd -G -vv [2026-10-16T13:12:47.129] debug: Log file re-opened [2026-10-16T13:12:47.130] debug: Reading cgroup.conf file /etc/slurm/cgroup.conf [2026-10-16T13:12:47.130] warning: cgroup/v1 plugin is deprecated, please upgrade to cgroup/v2 at your earliest convenience [2026-10-16T13:12:47.130] debug: cgroup/v1: init: Cgroup v1 plugin loaded [2026-10-16T13:12:47.131] debug: auth/munge: init: loaded [2026-10-16T13:12:47.131] debug: hash/k12: init: init: KangarooTwelve hash plugin loaded [2026-10-16T13:12:47.131] debug: tls/none: init: tls/none loaded [2026-10-16T13:12:47.135] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:12:47.135] GRES: Global AutoDetect=nvml(1) [2026-10-16T13:12:47.135] debug: gres/gpu: init: loaded [2026-10-16T13:12:47.135] debug: gres/shard: init: loaded [2026-10-16T13:12:47.136] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:12:47.144] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:12:47.206] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:12:47.211] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:12:47.217] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:12:47.224] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:12:47.226] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:12:47.231] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:12:47.232] Gres GPU plugin: Merging configured GRES with system GPUs [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] debug: gres/gpu: _merge_system_gres_conf: Including the following GPU matched between system and configuration: [2026-10-16T13:12:47.232] debug: GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] Gres GPU plugin: Final merged GRES list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: gpu device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: gpu device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] gres/shard: gres_c_s_init_share_devices: GRES: Initialized gres.conf list [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia1 major 195, minor 1 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia3 major 195, minor 3 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia4 major 195, minor 4 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia2 major 195, minor 2 [2026-10-16T13:12:47.232] GRES: _set_gres_device_desc : /dev/nvidia0 major 195, minor 0 [2026-10-16T13:12:47.232] GRES: shard device number 1(/dev/nvidia1):c 195:1 rwm [2026-10-16T13:12:47.232] GRES: shard device number 3(/dev/nvidia3):c 195:3 rwm [2026-10-16T13:12:47.232] GRES: shard device number 4(/dev/nvidia4):c 195:4 rwm [2026-10-16T13:12:47.232] GRES: shard device number 2(/dev/nvidia2):c 195:2 rwm [2026-10-16T13:12:47.232] GRES: shard device number 0(/dev/nvidia0):c 195:0 rwm [2026-10-16T13:12:47.232] Final gres.conf list: [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_5090 Count:1 Cores(16):0-7 Links:-1,0,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia1 UniqueId:GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,-1,0,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia3 UniqueId:GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,-1,0,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia4 UniqueId:GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:12:47.232] GRES[gpu] Type:a100 Count:1 Cores(16):0-7 Links:0,0,0,-1,0 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia2 UniqueId:GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:12:47.232] GRES[gpu] Type:rtx_3080 Count:1 Cores(16):0-7 Links:0,0,0,0,-1 Flags:HAS_FILE,HAS_TYPE,ENV_NVML File:/dev/nvidia0 UniqueId:GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_5090 Count=1 Index=1 ID=7696487 File=/dev/nvidia1 Cores=0-7 CoreCnt=16 Links=-1,0,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=3 ID=7696487 File=/dev/nvidia3 Cores=0-7 CoreCnt=16 Links=0,-1,0,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=4 ID=7696487 File=/dev/nvidia4 Cores=0-7 CoreCnt=16 Links=0,0,-1,0,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=a100 Count=1 Index=2 ID=7696487 File=/dev/nvidia2 Cores=0-7 CoreCnt=16 Links=0,0,0,-1,0 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.232] Gres Name=gpu Type=rtx_3080 Count=1 Index=0 ID=7696487 File=/dev/nvidia0 Cores=0-7 CoreCnt=16 Links=0,0,0,0,-1 Flags=HAS_FILE,HAS_TYPE,ENV_NVML [2026-10-16T13:12:47.241] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin # /opt/slurm/sbin/slurmd -C -vvv [2026-10-16T13:13:23.840] debug2: hwloc_topology_init [2026-10-16T13:13:23.840] debug2: hwloc_topology_load [2026-10-16T13:13:23.844] debug2: _check_full_access: got full access to the cpuset topology [2026-10-16T13:13:23.844] debug: CPUs:16 Boards:1 Sockets:2 CoresPerSocket:8 ThreadsPerCore:1 [2026-10-16T13:13:23.844] debug: gpu/nvml: init: init: GPU NVML plugin loaded [2026-10-16T13:13:23.851] debug2: gpu/nvml: _nvml_init: Successfully initialized NVML [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: Systems Graphics Driver Version: 615.71.09 [2026-10-16T13:13:23.851] debug: gpu/nvml: _get_system_gpu_list_nvml: NVML Library Version: 13.615.71.09 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVML API Version: 13 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Total CPU count: 16 [2026-10-16T13:13:23.851] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device count: 5 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 0: [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_5090 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-a17203cf-e03e-9bc8-237a-44302a1275e8 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:26:0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1A:00.0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: -1,0,0,0,0 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia1 [2026-10-16T13:13:23.913] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 0 is different from minor number 1 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.913] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.917] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.917] debug2: ------------------------------- [2026-10-16T13:13:23.917] debug2: *14001 MHz [0] [2026-10-16T13:13:23.917] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.917] debug2: --------------------------------- [2026-10-16T13:13:23.917] debug2: *3090 MHz [0] [2026-10-16T13:13:23.917] debug2: *3082 MHz [1] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *1635 MHz [194] [2026-10-16T13:13:23.917] debug2: ... [2026-10-16T13:13:23.917] debug2: *187 MHz [387] [2026-10-16T13:13:23.917] debug2: *180 MHz [388] [2026-10-16T13:13:23.917] debug2: *13801 MHz [1] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *7001 MHz [2] [2026-10-16T13:13:23.918] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.918] debug2: --------------------------------- [2026-10-16T13:13:23.918] debug2: *3090 MHz [0] [2026-10-16T13:13:23.918] debug2: *3082 MHz [1] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *1635 MHz [194] [2026-10-16T13:13:23.918] debug2: ... [2026-10-16T13:13:23.918] debug2: *187 MHz [387] [2026-10-16T13:13:23.918] debug2: *180 MHz [388] [2026-10-16T13:13:23.918] debug2: *810 MHz [3] [2026-10-16T13:13:23.919] debug2: Possible GPU Graphics Frequencies (389): [2026-10-16T13:13:23.919] debug2: --------------------------------- [2026-10-16T13:13:23.919] debug2: *3090 MHz [0] [2026-10-16T13:13:23.919] debug2: *3082 MHz [1] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *1635 MHz [194] [2026-10-16T13:13:23.919] debug2: ... [2026-10-16T13:13:23.919] debug2: *187 MHz [387] [2026-10-16T13:13:23.919] debug2: *180 MHz [388] [2026-10-16T13:13:23.919] debug2: *405 MHz [4] [2026-10-16T13:13:23.920] debug2: Possible GPU Graphics Frequencies (95): [2026-10-16T13:13:23.920] debug2: --------------------------------- [2026-10-16T13:13:23.920] debug2: *885 MHz [0] [2026-10-16T13:13:23.920] debug2: *877 MHz [1] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *532 MHz [47] [2026-10-16T13:13:23.920] debug2: ... [2026-10-16T13:13:23.920] debug2: *187 MHz [93] [2026-10-16T13:13:23.920] debug2: *180 MHz [94] [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 1: [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-cbe1156c-02a6-f112-7190-97a0b312e6bd [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:27:0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1B:00.0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,-1,0,0,0 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia3 [2026-10-16T13:13:23.921] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 1 is different from minor number 3 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.921] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.925] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.925] debug2: ------------------------------- [2026-10-16T13:13:23.925] debug2: *9501 MHz [0] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *9251 MHz [1] [2026-10-16T13:13:23.925] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.925] debug2: --------------------------------- [2026-10-16T13:13:23.925] debug2: *2100 MHz [0] [2026-10-16T13:13:23.925] debug2: *2085 MHz [1] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *1155 MHz [63] [2026-10-16T13:13:23.925] debug2: ... [2026-10-16T13:13:23.925] debug2: *225 MHz [125] [2026-10-16T13:13:23.925] debug2: *210 MHz [126] [2026-10-16T13:13:23.925] debug2: *5001 MHz [2] [2026-10-16T13:13:23.926] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.926] debug2: --------------------------------- [2026-10-16T13:13:23.926] debug2: *2100 MHz [0] [2026-10-16T13:13:23.926] debug2: *2085 MHz [1] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *1155 MHz [63] [2026-10-16T13:13:23.926] debug2: ... [2026-10-16T13:13:23.926] debug2: *225 MHz [125] [2026-10-16T13:13:23.926] debug2: *210 MHz [126] [2026-10-16T13:13:23.926] debug2: *810 MHz [3] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *2100 MHz [0] [2026-10-16T13:13:23.927] debug2: *2085 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *1155 MHz [63] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [125] [2026-10-16T13:13:23.927] debug2: *210 MHz [126] [2026-10-16T13:13:23.927] debug2: *405 MHz [4] [2026-10-16T13:13:23.927] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.927] debug2: --------------------------------- [2026-10-16T13:13:23.927] debug2: *420 MHz [0] [2026-10-16T13:13:23.927] debug2: *405 MHz [1] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *315 MHz [7] [2026-10-16T13:13:23.927] debug2: ... [2026-10-16T13:13:23.927] debug2: *225 MHz [13] [2026-10-16T13:13:23.927] debug2: *210 MHz [14] [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 2: [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-f997bc4f-a808-7edf-5a09-fc4b2f638595 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:30:0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:1E:00.0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,-1,0,0 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia4 [2026-10-16T13:13:23.928] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 2 is different from minor number 4 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.928] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.932] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.932] debug2: ------------------------------- [2026-10-16T13:13:23.932] debug2: *9501 MHz [0] [2026-10-16T13:13:23.932] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.932] debug2: --------------------------------- [2026-10-16T13:13:23.932] debug2: *2100 MHz [0] [2026-10-16T13:13:23.932] debug2: *2085 MHz [1] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *1155 MHz [63] [2026-10-16T13:13:23.932] debug2: ... [2026-10-16T13:13:23.932] debug2: *225 MHz [125] [2026-10-16T13:13:23.932] debug2: *210 MHz [126] [2026-10-16T13:13:23.932] debug2: *9251 MHz [1] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *5001 MHz [2] [2026-10-16T13:13:23.933] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.933] debug2: --------------------------------- [2026-10-16T13:13:23.933] debug2: *2100 MHz [0] [2026-10-16T13:13:23.933] debug2: *2085 MHz [1] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *1155 MHz [63] [2026-10-16T13:13:23.933] debug2: ... [2026-10-16T13:13:23.933] debug2: *225 MHz [125] [2026-10-16T13:13:23.933] debug2: *210 MHz [126] [2026-10-16T13:13:23.933] debug2: *810 MHz [3] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *2100 MHz [0] [2026-10-16T13:13:23.934] debug2: *2085 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *1155 MHz [63] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [125] [2026-10-16T13:13:23.934] debug2: *210 MHz [126] [2026-10-16T13:13:23.934] debug2: *405 MHz [4] [2026-10-16T13:13:23.934] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.934] debug2: --------------------------------- [2026-10-16T13:13:23.934] debug2: *420 MHz [0] [2026-10-16T13:13:23.934] debug2: *405 MHz [1] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *315 MHz [7] [2026-10-16T13:13:23.934] debug2: ... [2026-10-16T13:13:23.934] debug2: *225 MHz [13] [2026-10-16T13:13:23.934] debug2: *210 MHz [14] [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 3: [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_a100_80gb_pcie [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-418da940-1928-71e6-fa74-8889e551d850 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:63:0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:3F:00.0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,-1,0 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia2 [2026-10-16T13:13:23.936] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 3 is different from minor number 2 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.936] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.937] debug2: Possible GPU Memory Frequencies (1): [2026-10-16T13:13:23.937] debug2: ------------------------------- [2026-10-16T13:13:23.937] debug2: *1512 MHz [0] [2026-10-16T13:13:23.938] debug2: Possible GPU Graphics Frequencies (81): [2026-10-16T13:13:23.938] debug2: --------------------------------- [2026-10-16T13:13:23.938] debug2: *1410 MHz [0] [2026-10-16T13:13:23.938] debug2: *1395 MHz [1] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *810 MHz [40] [2026-10-16T13:13:23.938] debug2: ... [2026-10-16T13:13:23.938] debug2: *225 MHz [79] [2026-10-16T13:13:23.938] debug2: *210 MHz [80] [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: GPU index 4: [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Name: nvidia_geforce_rtx_3080 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: UUID: GPU-6a11cc78-b0ab-cfa6-a6a0-cc5b220512b3 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Domain/Bus/Device: 0:64:0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: PCI Bus ID: 00000000:40:00.0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: NVLinks: 0,0,0,0,-1 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Device File (minor number): /dev/nvidia0 [2026-10-16T13:13:23.938] debug: gpu/nvml: _get_system_gpu_list_nvml: Note: GPU index 4 is different from minor number 0 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: CPU Affinity Range - Machine: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: Core Affinity Range - Abstract: 0-7 [2026-10-16T13:13:23.938] debug2: gpu/nvml: _get_system_gpu_list_nvml: MIG mode: disabled [2026-10-16T13:13:23.942] debug2: Possible GPU Memory Frequencies (5): [2026-10-16T13:13:23.942] debug2: ------------------------------- [2026-10-16T13:13:23.942] debug2: *9501 MHz [0] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *9251 MHz [1] [2026-10-16T13:13:23.943] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.943] debug2: --------------------------------- [2026-10-16T13:13:23.943] debug2: *2100 MHz [0] [2026-10-16T13:13:23.943] debug2: *2085 MHz [1] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *1155 MHz [63] [2026-10-16T13:13:23.943] debug2: ... [2026-10-16T13:13:23.943] debug2: *225 MHz [125] [2026-10-16T13:13:23.943] debug2: *210 MHz [126] [2026-10-16T13:13:23.943] debug2: *5001 MHz [2] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *810 MHz [3] [2026-10-16T13:13:23.944] debug2: Possible GPU Graphics Frequencies (127): [2026-10-16T13:13:23.944] debug2: --------------------------------- [2026-10-16T13:13:23.944] debug2: *2100 MHz [0] [2026-10-16T13:13:23.944] debug2: *2085 MHz [1] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *1155 MHz [63] [2026-10-16T13:13:23.944] debug2: ... [2026-10-16T13:13:23.944] debug2: *225 MHz [125] [2026-10-16T13:13:23.944] debug2: *210 MHz [126] [2026-10-16T13:13:23.944] debug2: *405 MHz [4] [2026-10-16T13:13:23.945] debug2: Possible GPU Graphics Frequencies (15): [2026-10-16T13:13:23.945] debug2: --------------------------------- [2026-10-16T13:13:23.945] debug2: *420 MHz [0] [2026-10-16T13:13:23.945] debug2: *405 MHz [1] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *315 MHz [7] [2026-10-16T13:13:23.945] debug2: ... [2026-10-16T13:13:23.945] debug2: *225 MHz [13] [2026-10-16T13:13:23.945] debug2: *210 MHz [14] [2026-10-16T13:13:23.945] gpu/nvml: _get_system_gpu_list_nvml: 5 GPU system device(s) detected [2026-10-16T13:13:23.954] debug2: gpu/nvml: _nvml_shutdown: Successfully shut down NVML [2026-10-16T13:13:23.954] debug: gpu/nvml: fini: fini: unloading GPU NVML plugin [2026-10-16T13:13:23.955] Configured with rsmi, but rsmi isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] Configured with oneAPI, but oneAPI isn't enabled during the build. [2026-10-16T13:13:23.955] debug: gpu/generic: init: init: GPU Generic plugin loaded [2026-10-16T13:13:23.955] debug: gpu/generic: fini: fini: unloading GPU Generic plugin [2026-10-16T13:13:23.955] debug: gpu/nrt: init: init: GPU NRT plugin loaded [2026-10-16T13:13:23.955] error: System GPU detection failed [2026-10-16T13:13:23.955] debug: gpu/nrt: fini: fini: unloading GPU NRT plugin NodeName=moose66 CPUs=16 Boards=1 SocketsPerBoard=2 CoresPerSocket=8 ThreadsPerCore=1 RealMemory=385100 Gres=gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 Found gpu:nvidia_geforce_rtx_5090:1,gpu:nvidia_geforce_rtx_3080:3,gpu:nvidia_a100_80gb_pcie:1 with Autodetect=nvml (Substring of gpu name may be used instead) UpTime=0-03:12:06 -Dj