<div dir="ltr"><div><br></div>I am running a slurm client on a virtual machine. The virtual machine originally had a core count of 10. But I have now increased the cores to 16, but "slurmd -C" continues to show 10. I have increased the core count in the slurm.conf file. and that is being seen correctly. The state of the node is stuck in a Drain state because of this conflict. How do I get slurmd -C to see the new number of cores?<div><br></div><div>I'm running slurm 18.08. I have tried running "scontrol reconfigure" on the head node. I have restarted slurmd on all the client nodes, and I have restarted slurmctld on the master node.<br><div><br></div><div>Where is the data about compute note CPUs stored? I can't seem to find a config or setting file on the compute node.</div><div><br></div><div>The compute node that I am working on is "liverpool"</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(2,166,1);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><b>mtie@liverpool</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> slurmd -C</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NodeName=liverpool CPUs=10 Boards=1 SocketsPerBoard=10 CoresPerSocket=1 ThreadsPerCore=1 RealMemory=64263</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">UpTime=1-21:55:36</span></p><p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(2,166,1);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><b>mtie@liverpool</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)"> lscpu</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Architecture:<span class="gmail-Apple-converted-space"> </span>x86_64</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">CPU op-mode(s):<span class="gmail-Apple-converted-space"> </span>32-bit, 64-bit</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Byte Order:<span class="gmail-Apple-converted-space"> </span>Little Endian</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">CPU(s):<span class="gmail-Apple-converted-space"> </span>16</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">On-line CPU(s) list: <span class="gmail-Apple-converted-space"> </span>0-15</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Thread(s) per core:<span class="gmail-Apple-converted-space"> </span>1</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Core(s) per socket:<span class="gmail-Apple-converted-space"> </span>4</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Socket(s): <span class="gmail-Apple-converted-space"> </span>4</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NUMA node(s):<span class="gmail-Apple-converted-space"> </span>1</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Vendor ID: <span class="gmail-Apple-converted-space"> </span>GenuineIntel</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">CPU family:<span class="gmail-Apple-converted-space"> </span>15</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Model: <span class="gmail-Apple-converted-space"> </span>6</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Model name:<span class="gmail-Apple-converted-space"> </span>Common KVM processor</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Stepping:<span class="gmail-Apple-converted-space"> </span>1</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">CPU MHz: <span class="gmail-Apple-converted-space"> </span>2600.028</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">BogoMIPS:<span class="gmail-Apple-converted-space"> </span>5200.05</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Hypervisor vendor: <span class="gmail-Apple-converted-space"> </span>KVM</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Virtualization type: <span class="gmail-Apple-converted-space"> </span>full</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">L1d cache: <span class="gmail-Apple-converted-space"> </span>32K</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">L1i cache: <span class="gmail-Apple-converted-space"> </span>32K</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">L2 cache:<span class="gmail-Apple-converted-space"> </span>4096K</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">L3 cache:<span class="gmail-Apple-converted-space"> </span>16384K</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">NUMA node0 CPU(s): <span class="gmail-Apple-converted-space"> </span>0-15</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Flags: <span class="gmail-Apple-converted-space"> </span>fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm</span></p><p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> more /etc/slurm/slurm.conf | grep liverpool</span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">NodeName=</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(178,38,20)"><b>liverpool</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> NodeAddr=137.22.10.202 CPUs=16 State=UNKNOWN</span></p><p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">PartitionName=BioSlurm Nodes=</span><span class="gmail-s4" style="font-variant-ligatures:no-common-ligatures;color:rgb(178,38,20)"><b>liverpool</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>Default=YES MaxTime=INFINITE State=UP</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> sinfo -n liverpool -o %c</span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">CPUS</span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">16</span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures;color:rgb(2,166,1)"><b>mtie@liverpool</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(68,37,213)"><b> ~ $</b></span><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"> sinfo -n liverpool -o %E</span></p><p class="gmail-p1" style="margin:0px;font:17px Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">REASON</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures">Low socket*core*thread count, Low CPUs</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:17px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);background-color:rgb(227,222,190)"><span class="gmail-s3" style="font-variant-ligatures:no-common-ligatures"><br></span></p><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br>Any advice?</div><div><br></div><div><br></div></div></div></div></div></div></div></div></div></div></div>