<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Hello all,</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"></p>
<div>I'm trying to turn off core specialization in my cluster by setting CoreSpecCount=0, but checking with scontrol does not show my changes. If I set CoreSpec=1 or CoreSpecCount=2, or anything except 0, the changes are applied correctly. But when I set it
to 0, no change is applied -- it remains on whatever the previous number was.<br>
<br>
with CoreSpecCount=1:<br>
<br>
---------------------------------------<br>
# scontrol show node node016<br>
NodeName=node016 Arch=x86_64 CoresPerSocket=18⋅<br>
CPUAlloc=0 CPUTot=72 CPULoad=0.01<br>
AvailableFeatures=(null)<br>
ActiveFeatures=(null)<br>
Gres=(null)<br>
NodeAddr=node016 NodeHostName=node016⋅<br>
OS=Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018⋅<br>
RealMemory=95306 AllocMem=0 FreeMem=N/A Sockets=2 Boards=1<br>
CoreSpecCount=1 CPUSpecList=70-71⋅<br>
State=IDLE ThreadsPerCore=2 TmpDisk=2038 Weight=1 Owner=N/A MCS_label=N/A<br>
Partitions=test⋅<br>
BootTime=2019-06-19T08:41:49 SlurmdStartTime=2019-06-27T09:06:26<br>
CfgTRES=cpu=72,mem=95306M,billing=72<br>
AllocTRES=<br>
CapWatts=n/a<br>
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0<br>
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>
---------------------------------------<br>
<br>
That is correct.<br>
<br>
with CoreSpecCount=0:<br>
<br>
---------------------------------------<br>
# scontrol show node node016<br>
NodeName=node016 Arch=x86_64 CoresPerSocket=18⋅<br>
CPUAlloc=0 CPUTot=72 CPULoad=0.01<br>
AvailableFeatures=(null)<br>
ActiveFeatures=(null)<br>
Gres=(null)<br>
NodeAddr=node016 NodeHostName=node016⋅<br>
OS=Linux 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018⋅<br>
RealMemory=95306 AllocMem=0 FreeMem=92773 Sockets=2 Boards=1<br>
CoreSpecCount=1 CPUSpecList=70-71⋅<br>
State=IDLE ThreadsPerCore=2 TmpDisk=2038 Weight=1 Owner=N/A MCS_label=N/A<br>
Partitions=test⋅<br>
BootTime=2019-06-19T08:41:49 SlurmdStartTime=2019-06-27T09:06:26<br>
CfgTRES=cpu=72,mem=95306M,billing=72<br>
AllocTRES=<br>
CapWatts=n/a<br>
CurrentWatts=0 LowestJoules=0 ConsumedJoules=0<br>
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s<br>
---------------------------------------<br>
<br>
That is wrong. It's exactly the same -- CoreSpecCount still shows 1.<br>
<br>
The weird thing is that if I run slurmd in the foreground in verbose mode on the node with "slurmd -cDvvf /etc/slurm/slurm.conf", the change appears to be recognized.<br>
<br>
Results with CoreSpecCount=1:<br>
</div>
<div><br>
</div>
<div>
<div>---------------------------------------<br>
slurmd: got reconfigure request<br>
slurmd: all threads complete<br>
slurmd: debug: Reading slurm.conf file: /etc/slurm/slurm.conf<br>
slurmd: debug: Ignoring obsolete CacheGroups option.<br>
slurmd: debug: Log file re-opened<br>
slurmd: debug: CPUs:72 Boards:1 Sockets:2 CoresPerSocket:18 ThreadsPerCore:2<br>
slurmd: Message aggregation disabled<br>
slurmd: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<br>
slurmd: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<br>
slurmd: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<br>
slurmd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/cpuset/slurm' already exists<br>
slurmd: debug: xcgroup_instantiate: cgroup '/sys/fs/cgroup/cpuset/slurm/system' already exists<br>
slurmd: debug: system cgroup: system cpuset cgroup initialized<br>
slurmd: Resource spec: Reserved abstract CPU IDs: 70-71<br>
slurmd: Resource spec: Reserved machine CPU IDs: 35,71<br>
slurmd: debug: Resource spec: Reserved system memory limit not configured for this node<br>
---------------------------------------<br>
<br>
Results with CoreSpecCount=0:<br>
<br>
---------------------------------------<br>
slurmd: got reconfigure request<br>
slurmd: all threads complete<br>
slurmd: debug: Reading slurm.conf file: /etc/slurm/slurm.conf<br>
slurmd: debug: Ignoring obsolete CacheGroups option.<br>
slurmd: debug: Log file re-opened<br>
slurmd: debug: CPUs:72 Boards:1 Sockets:2 CoresPerSocket:18 ThreadsPerCore:2<br>
slurmd: Message aggregation disabled<br>
slurmd: debug: Reading cgroup.conf file /etc/slurm/cgroup.conf<br>
slurmd: debug: Resource spec: No specialized cores configured by default on this node<br>
slurmd: debug: Resource spec: Reserved system memory limit not configured for this node<br>
---------------------------------------<br>
<br>
The reserved CPUs have been removed as they should be. So why does scontrol still show the incorrect value (and jobs still do not run on those cores)?<br>
<br>
Dave<br>
<br>
</div>
<br>
</div>
<font color="#008080"><i>David Guertin</i></font>
<p></p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div><font color="#008080"><i>Information Technology Services</i></font></div>
<div><font color="#008080"><i>Middlebury College</i></font></div>
<div><font color="#008080"><i>700 Exchange St.</i></font></div>
<div><font color="#008080"><i>Middlebury, VT 05753</i></font></div>
<div><font color="#008080"><i>(802)443-3143</i></font></div>
</div>
</div>
</div>
</body>
</html>