<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi all - I’m new to slurm, and in many ways it’s been very nice to work with, but I’m having an issue trying to properly set up thread/core/socket counts on nodes. Basically, if I don’t specify anything except CPUs, the node is available, but doesn’t appear to know about cores and hyperthreading. If I do try to specify that info it claims that the numbers aren’t consistent and sets the node to drain.<div class=""><br class=""></div><div class="">This is all on CentOS 7, slurm 18.08, and FastSchedule is set to 0.</div><div class=""><br class=""></div><div class="">First type of node, 2 x 8 core CPUs, hyperthreading on, nothing specified in slurm.conf except CPUs. /proc/cpuinfo confirms that there are 32 “cpus”, with the expected values for physical id and core id.</div><div class=""><br class=""></div><div class="">From slurm.conf<br class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">NodeName=compute-2-0 NodeAddr=10.1.255.250 CPUs=32 Weight=20511700 Feature=rack-2,32CPUs</div><div class=""><br class=""></div></blockquote>from scontrol show node<br class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class="">NodeName=compute-2-0 Arch=x86_64 CoresPerSocket=1</div><div class=""> CPUAlloc=0 CPUTot=32 CPULoad=0.04</div><div class=""> AvailableFeatures=rack-2,32CPUs</div><div class=""> ActiveFeatures=rack-2,32CPUs</div><div class=""> Gres=(null)</div><div class=""> NodeAddr=10.1.255.250 NodeHostName=compute-2-0 Version=18.08</div><div class=""> OS=Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018</div><div class=""> RealMemory=257742 AllocMem=0 FreeMem=255703 Sockets=32 Boards=1</div><div class=""> State=IDLE ThreadsPerCore=1 TmpDisk=913567 Weight=20511700 Owner=N/A MCS_label=N/A</div><div class=""> Partitions=CLUSTER,WHEEL,n2013f</div><div class=""> BootTime=2018-10-10T11:06:42 SlurmdStartTime=2018-10-10T11:07:16</div><div class=""> CfgTRES=cpu=32,mem=257742M,billing=94</div><div class=""> AllocTRES=</div><div class=""> CapWatts=n/a</div><div class=""> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</div><div class=""> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</div></div><div class=""><br class=""></div></blockquote><br class=""></div><div class="">Second type of node, 2 x 4 core CPUs, hyperthreading on. /proc/cpuinfo confirms that there are 16 “cpus”, with the expected values for physical id and core id.</div><div class=""><br class=""></div><div class="">If I set the numbers of sockets/cores/threads as I think is correct (note that this is a different type of machine than the previous), </div><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class="">NodeName=compute-0-0 NodeAddr=10.1.255.253 CPUs=16 Weight=20495900 Feature=rack-0,16CPUs Sockets=2 CoresPerSocket=4 ThreadsPerCore=2</div></blockquote>I get the following<br class=""><blockquote style="margin: 0 0 0 40px; border: none; padding: 0px;" class=""><div class=""><div class="">NodeName=compute-0-0 Arch=x86_64 CoresPerSocket=4</div><div class=""> CPUAlloc=0 CPUTot=16 CPULoad=0.42</div><div class=""> AvailableFeatures=rack-0,16CPUs</div><div class=""> ActiveFeatures=rack-0,16CPUs</div><div class=""> Gres=(null)</div><div class=""> NodeAddr=10.1.255.253 NodeHostName=compute-0-0 Version=18.08</div><div class=""> OS=Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018</div><div class=""> RealMemory=11842 AllocMem=0 FreeMem=11335 Sockets=2 Boards=1</div><div class=""> State=IDLE+DRAIN ThreadsPerCore=2 TmpDisk=275125 Weight=20495900 Owner=N/A MCS_label=N/A</div><div class=""> Partitions=CLUSTER,WHEEL,ib_qdr</div><div class=""> BootTime=2018-10-10T11:06:55 SlurmdStartTime=2018-10-10T11:07:34</div><div class=""> CfgTRES=cpu=16,mem=11842M,billing=18</div><div class=""> AllocTRES=</div><div class=""> CapWatts=n/a</div><div class=""> CurrentWatts=0 LowestJoules=0 ConsumedJoules=0</div><div class=""> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</div><div class=""> Reason=Low socket*core count [root@2018-10-10T10:26:14]</div></div></blockquote><br class=""><div class="">I feel like there are couple of things that are suspicious, but I’m not sure</div><div class="">1. I get the impression that slurm is supposed to be able to automatically figure out the architecture of the node, but in the first example there’s no evidence of that in the scontrol output. </div><div class="">2. When I set the various architecture related parameters it claims that the numbers don’t match, even though sockets*cores*threads = 2*4*2 = 16 = CPUs</div><div class=""><br class=""></div><div class="">Does anyone have any idea as to what’s going on, or what other information would be useful for debugging?</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>thanks,</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Noam</div></body></html>