<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We've added a new GPU node to our cluster with 32 cores. It contains 2 16-core sockets, and hyperthreading is turned off, so the total is 32 cores. But jobs are only being allowed to use 16 cores.<br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Here's the relevant line from slurm.conf:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
NodeName=node020 CoresPerSocket=16 RealMemory=257600 ThreadsPerCore=1 Boards=1 SocketsPerBoard=2 Weight=100 Gres=gpu:rtxa5000:4<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
And here's scontrol output for the node. Note that even though CPUTot=32, CfgTRES=cpu=16 instead of 32:<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
# scontrol show node node020
<div>NodeName=node020 Arch=x86_64 CoresPerSocket=16 </div>
<div> CPUAlloc=16 CPUTot=32 CPULoad=7.29</div>
<div> AvailableFeatures=(null)</div>
<div> ActiveFeatures=(null)</div>
<div> Gres=gpu:rtxa5000:4</div>
<div> NodeAddr=node020 NodeHostName=node020 Version=19.05.8</div>
<div> OS=Linux 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 </div>
<div> RealMemory=257600 AllocMem=126976 FreeMem=1393 Sockets=2 Boards=1</div>
<div> State=MIXED ThreadsPerCore=1 TmpDisk=2038 Weight=100 Owner=N/A MCS_label=N/A</div>
<div> Partitions=gpu-long,gpu-short,gpu-standard </div>
<div> BootTime=2022-04-05T11:37:08 SlurmdStartTime=2022-04-05T11:43:00</div>
<div> CfgTRES=cpu=16,mem=257600M,billing=16,gres/gpu=4</div>
<div> AllocTRES=cpu=16,mem=124G,gres/gpu=2</div>
<div> CapWatts=n/a</div>
<div> CurrentWatts=0 AveWatts=0</div>
<span> ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Why isn't this node allocating all 32 cores?<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,<br>
</div>
<div id="Signature">
<div>
<div></div>
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font color="#008080"><span style="color:rgb(0,0,0)">David Guertin</span></font> </div>
</div>
</div>
</div>
</body>
</html>