[slurm-users] 4 sockets but "

[Diego Zuccato]

Il 20/07/2021 13:23, Ole Holm Nielsen ha scritto:

Hello Ole.

> The Xeon Platinum 8268 is a 24-core CPU:
> https://ark.intel.com/content/www/us/en/ark/products/192481/intel-xeon-platinum-8268-processor-35-75m-cache-2-90-ghz.html 
Yup.

> 1. So you have 4 physical sockets in each node?
Correct.

> 2. Did you define a Sub NUMA Cluster (SNC) BIOS setting?  Then each 
> physical socket would show up as two sockets (memory controllers), for a 
> total of 8 "sockets" in your 4-socket system.
I don't think so. Unless that's the default, I didn't change anything in 
the BIOS. Just checked the second of the three nodes (still no SO 
installed) and found it under Chipset Configuration -> North bridge -> 
UPI configuration -> SNC : it's set to Disable.
> 3. Which Slurm version are you running, and which OS version?
Slurm 18.08.5, standard packages from Debian stable.

> 4. What is the output of "slurmd -C" (Print actual hardware 
> configuration) on the node?
root at str957-mtx-20:~# slurmd -C
NodeName=str957-mtx-20 CPUs=192 Boards=1 SocketsPerBoard=4 
CoresPerSocket=24 ThreadsPerCore=2 RealMemory=1160347
UpTime=1-01:50:48
So the node seems to correctly recognizes the underlying HW.

I tried to copy&paste "SocketsPerBoard=4 CoresPerSocket=24 
ThreadsPerCore=2 RealMemory=1160347" in the nodes definition line (the 
only difference being SocketsPerBoard instead of Sockets), but the 
result is always the same.

> Maybe my Wiki notes could be helpful?
> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#compute-node-configuration 
Tks. Interesting, but I don't se pam_slurm_adopt. Other than that, it 
seems very much like what I'm doing.

BYtE,
  Diego

> On 7/20/21 12:49 PM, Diego Zuccato wrote:
>> Hello all.
>>
>> It's been since yesterday that I'm facing this issue.
>> I'm configuring 3 new quad-socket nodes defined as:
>> NodeName=str957-mtx-[20-22] Sockets=4 CoresPerSocket=24 \
>>     RealMemory=1160347 Weight=8 Feature=ib,matrix,intel,avx
>>
>> But scontrol show node str957-mtx-20 reports:
>> NodeName=str957-mtx-20 CoresPerSocket=1
>> [...]
>>     RealMemory=1160347 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1
>>     MemSpecLimit=2048
>>     State=DOWN*+DRAIN ThreadsPerCore=2 TmpDisk=0 Weight=8 Owner=N/A 
>> MCS_label=N/A
>>     Partitions=m3
>>     BootTime=None SlurmdStartTime=None
>>     CfgTRES=cpu=4,mem=1160347M,billing=2783
>>     AllocTRES=
>>     CapWatts=n/a
>>     CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
>>     ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
>>     Reason=Low socket*core*thread count, Low CPUs
>>
>> The only thing that's right is that ThreadsPerCore=2 ...
>> The last block from "cat /proc/cpuinfo" on the node reports:
>> processor       : 191
>> vendor_id       : GenuineIntel
>> cpu family      : 6
>> model           : 85
>> model name      : Intel(R) Xeon(R) Platinum 8268 CPU @ 2.90GHz
>> stepping        : 7
>> microcode       : 0x5003003
>> cpu MHz         : 1200.331
>> cache size      : 36608 KB
>> physical id     : 3
>> siblings        : 48
>> core id         : 29
>> cpu cores       : 24
>> apicid          : 251
>> initial apicid  : 251
>> fpu             : yes
>> fpu_exception   : yes
>> cpuid level     : 22
>> wp              : yes
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe 
>> syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts 
>> rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq 
>> dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm 
>> pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes 
>> xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 
>> cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp 
>> ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase 
>> tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f 
>> avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw 
>> avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc 
>> cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke 
>> avx512_vnni md_clear flush_l1d arch_capabilities
>> bugs            : spectre_v1 spectre_v2 spec_store_bypass swapgs taa 
>> itlb_multihit
>> bogomips        : 5804.13
>> clflush size    : 64
>> cache_alignment : 64
>> address sizes   : 46 bits physical, 48 bits virtual
>> power management:
>>
>> I already tried changing the line to cheat (telling it there are only 
>> 2 sockets with 24 cores, thus reducing to half node) but nothing 
>> changed. I restarted slurmctld after every change in slurm.conf just 
>> to be sure.
>> Any idea?
> 

-- 
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786