<div dir="ltr"><div>To answer your direct question, the ground truth of 'slurmctld -C' is what the kernel thinks the hardware is (what you see in lscpu, except it probably employs some tricks for VMs with an odd topology). And it got severely confused by what the kernel reported to it. I know from experience that certain odd cloud VM shapes throw it off balance.<br></div><div><br></div><div>I do not really like the output of lscpu. I have never seen such a strange shape of a VM. CPU family 15 is in the Pentium 4 line <<a href="https://software.intel.com/en-us/articles/intel-architecture-and-processor-identification-with-cpuid-model-and-family-numbers">https://software.intel.com/en-us/articles/intel-architecture-and-processor-identification-with-cpuid-model-and-family-numbers</a>>, and model 6 was the last breath of this unsuccessful NetBurst architecture--such a rarity that Linux kernel does not even have in its database: "<span style="font-variant-ligatures:no-common-ligatures">Common KVM processor" is a slug for "everything else that one of these soul-sapping KVMs may return"</span>. Flags show that the processor supports SSE2 and 3, but not 4.1, 4.2 or AVX, which is consistent with a Pentium 4, but 16M of L3 cache is about an average total RAM in a desktop at the time P4 was a thing.
And the CPU is a NUMA (no real Pentium 4 had the NUMA, only SMP)¹. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Any advice? <br></div></blockquote><div><br></div><div>My best advice would be to either use a different hypervisor or tune correctly the one you have. Sometimes a hypervisor is tuned for live VM migration, when it is frozen on one hardware type and thawed on another, and may tweak the CPUID in advance to hide features from the guest OS so that it would be able to continue if migrated to less capable hardware; but still, using the P4 as the least common denominator is way too extreme. Something is seriously wrong on the KVM host.<br></div><div><br></div><div>The VM itself is braindead. Even if you will have got it up and running, the absence of SSE4.1 and 4.2, AVX, AXV2, and AVX512²
would make it about as efficient a computing node as a brick. Unless the host CPU is really a Presler Pentium 4, in which case you are way too long overdue for a hardware upgrade :)))</div><div></div><div><br></div><div> -kkm<br></div><div></div><div> ____</div><div>¹ It's not impossible that lscpu shows an SMP machine as if containing a single NUMA node, but I have a recollection that this is not the case. I haven't seen a non-NUMA CPU in quite a while.</div>² Intel had gone besides-itself-creative this time. It was even bigger a naming leap than switching from Roman to decimal between Pentium III to Pentium *drum roll* 4 *cymbal crash*.<br><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 8, 2020 at 1:20 PM mike tie <<a href="mailto:mtie@carleton.edu">mtie@carleton.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div>I am running a slurm client on a virtual machine. The virtual machine originally had a core count of 10. But I have now increased the cores to 16, but "slurmd -C" continues to show 10. I have increased the core count in the slurm.conf file. and that is being seen correctly. The state of the node is stuck in a Drain state because of this conflict. How do I get slurmd -C to see the new number of cores?<div><br></div><div>I'm running slurm 18.08. I have tried running "scontrol reconfigure" on the head node. I have restarted slurmd on all the client nodes, and I have restarted slurmctld on the master node.<br><div><br></div><div>Where is the data about compute note CPUs stored? I can't seem to find a config or setting file on the compute node.</div><div><br></div><div>The compute node that I am working on is "liverpool"</div><div><br></div><div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(2,166,1);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><b>mtie@liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> slurmd -C</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">NodeName=liverpool CPUs=10 Boards=1 SocketsPerBoard=10 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=64263</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">UpTime=1-21:55:36</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(2,166,1);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><b>mtie@liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> lscpu</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Architecture:<span> </span>x86_64</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">CPU op-mode(s):<span> </span>32-bit, 64-bit</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Byte Order:<span> </span>Little Endian</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">CPU(s):<span> </span>16</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">On-line CPU(s) list: <span> </span>0-15</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Thread(s) per core:<span> </span>1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Core(s) per socket:<span> </span>4</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Socket(s): <span> </span>4</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">NUMA node(s):<span> </span>1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Vendor ID: <span> </span>GenuineIntel</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">CPU family:<span> </span>15</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Model: <span> </span>6</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Model name:<span> </span>Common KVM processor</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Stepping:<span> </span>1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">CPU MHz: <span> </span>2600.028</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">BogoMIPS:<span> </span>5200.05</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Hypervisor vendor: <span> </span>KVM</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Virtualization type: <span> </span>full</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">L1d cache: <span> </span>32K</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">L1i cache: <span> </span>32K</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">L2 cache:<span> </span>4096K</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">L3 cache:<span> </span>16384K</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">NUMA node0 CPU(s): <span> </span>0-15</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Flags: <span> </span>fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span style="font-variant-ligatures:no-common-ligatures"> more /etc/slurm/slurm.conf | grep liverpool</span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">NodeName=</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(178,38,20)"><b>liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures"> NodeAddr=137.22.10.202 CPUs=16 State=UNKNOWN</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">PartitionName=BioSlurm Nodes=</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(178,38,20)"><b>liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Default=YES MaxTime=INFINITE State=UP</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span style="font-variant-ligatures:no-common-ligatures"> sinfo -n liverpool -o %c</span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">CPUS</span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">16</span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span style="font-variant-ligatures:no-common-ligatures"> sinfo -n liverpool -o %E</span></p><p style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">REASON</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures">Low socket*core*thread count, Low CPUs</span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></p><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br>Any advice?</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>