<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>It looks like you have hyper-threading turned on, but haven’t defined the ThreadsPerCore=2. You either need to turn off Hyper-threading in the BIOS or changed the definition of ThreadsPerCore in slurm.conf.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mike<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Robert Kudyba <rkudyba@fordham.edu><br><b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br><b>Date: </b>Thursday, April 23, 2020 at 08:27<br><b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br><b>Subject: </b>[External] [slurm-users] slurmd: error: Node configuration differs from hardware: CPUs=24:48(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw)<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div style='border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt'><p class=MsoNormal style='line-height:12.0pt;background:#FFEB9C'><b><span style='font-size:10.0pt;color:#9C6500'>CAUTION:</span></b><span style='font-size:10.0pt;color:black'> This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.<o:p></o:p></span></p></div><p class=MsoNormal><o:p> </o:p></p><div><div><div><p class=MsoNormal>Running Slurm 20.02 on Centos 7.7 on Bright Cluster 8.2. slurm.conf is on the head node. I don't see these errors on the other 2 nodes. After restarting slurmd on node003 I see this:<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>slurmd[400766]: error: Node configuration differs from hardware: CPUs=24:48(hw) Boards=1:1(hw) SocketsPerBoard=2:2(hw) CoresPerSocket=12:12(hw) ThreadsPerCore=1:2(hw)<br>Apr 23 10:05:49 node003 slurmd[400766]: Message aggregation disabled<br>Apr 23 10:05:49 node003 slurmd[400766]: CPU frequency setting not configured for this node<br>Apr 23 10:05:49 node003 slurmd[400770]: CPUs=24 Boards=1 Sockets=2 Cores=12 Threads=1 Memory=191880 TmpDisk=2038 Uptime=2488268 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)<br><br>From slurm.conf:<br># Nodes<br>NodeName=node[001-003] CoresPerSocket=12 RealMemory=191800 Sockets=2 Gres=gpu:v100:1<br># Partitions<br>$O Hidden=NO OverSubscribe=FORCE:12 GraceTime=0 PreemptMode=OFF ReqResv=NO AllowAccounts=ALL AllowQos=ALL LLN=NO ExclusiveUser=N$<br>PartitionName=gpuq Default=NO MinNodes=1 AllowGroups=ALL PriorityJobFactor=1 PriorityTier=1 DisableRootJobs=NO RootOnly=NO Hidde$<br># Generic resources types<br>GresTypes=gpu,mic<br>SelectType=select/cons_tres<br>SelectTypeParameters=CR_CPU<br>SchedulerTimeSlice=60<br>EnforcePartLimits=YES<br><br>lscpu<br>Architecture: x86_64<br>CPU op-mode(s): 32-bit, 64-bit<br>Byte Order: Little Endian<br>CPU(s): 48<br>On-line CPU(s) list: 0-47<br>Thread(s) per core: 2<br>Core(s) per socket: 12<br>Socket(s): 2<br>NUMA node(s): 2<br>Vendor ID: GenuineIntel<br>CPU family: 6<br>Model: 85<br>Model name: Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz<br>Stepping: 4<br>CPU MHz: 2600.000<br>BogoMIPS: 5200.00<br>Virtualization: VT-x<br>L1d cache: 32K<br>L1i cache: 32K<br>L2 cache: 1024K<br>L3 cache: 19712K<br>NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46<br>NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47<br><br>cat /etc/slurm/cgroup.conf| grep -v '#'<br>CgroupMountpoint="/sys/fs/cgroup"<br>CgroupAutomount=no<br>AllowedDevicesFile="/etc/slurm/cgroup_allowed_devices_file.conf"<br>TaskAffinity=no<br>ConstrainCores=no<br>ConstrainRAMSpace=no<br>ConstrainSwapSpace=no<br>ConstrainDevices=no<br>ConstrainKmemSpace=yes<br>AllowedRamSpace=100<br>AllowedSwapSpace=0<br>MinKmemSpace=30<br>MaxKmemPercent=100<br>MaxRAMPercent=100<br>MaxSwapPercent=100<br>MinRAMSpace=30<o:p></o:p></p></div><div><p class=MsoNormal>What else can I check?<o:p></o:p></p></div></div></div></div></body></html>