<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">Hi,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">We have Hyper-threading/SMT enabled on our cluster. It's challenging to fully utilize threads, as Brian suggests. We have a few workloads that benefit from it being enabled, but they
represent a minority of our overall workload. </span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">We use SelectTypeParameters=CR_Core_Memory. This configuration means that each thread counts as a CPU. The biggest issue with this has been user support. Most of our users are unaware
of SMT as a feature and often require consultation to ensure their resource requests match their workload. I'm invested in SMT at this point, but if I had a do over (in the same environment), I would disable Hyper-threading.</span></div>
<div><br>
</div>
<div><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">Ping me with questions. I'm happy to discuss our config and experience in greater detail.</span></div>
<div><br>
</div>
<div><span style="font-size: 12pt; font-family: Calibri, Helvetica, sans-serif;">Sebastian</span><br>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif;">
<div style="font-family:Tahoma; font-size:13px">
<div style="font-family:Tahoma; font-size:13px">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText">
<div id="emailSignature" style="display:block">--</div>
<div id="emailSignature" style="display:block"><br>
<table style="box-sizing: border-box; border-collapse: collapse; border-spacing: 0px; background-color: rgb(255, 255, 255); max-width: 100%; text-align: start; font-family: Helvetica, Arial, san-serif; width: 480px;">
<tbody style="box-sizing:border-box">
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box; vertical-align:top" width="75"><a href="http://www.unr.edu/" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; color: rgb(46, 108, 162); text-decoration: underline;"><img alt="University of Nevada, Reno" style="box-sizing:border-box; vertical-align:middle" width="75" src="https://www.unr.edu/assets/images/blockn-100x100.png"></a></td>
<td style="box-sizing:border-box" width="20"> </td>
<td style="box-sizing:border-box; vertical-align:top" width="385">
<table style="box-sizing:border-box; border-collapse:collapse; max-width:100%; width:385px; text-align:left">
<tbody style="box-sizing:border-box">
<tr style="box-sizing:border-box">
<td style="box-sizing:border-box; vertical-align:top"><span style="box-sizing:border-box"><span style="box-sizing: border-box; color: rgb(0, 51, 102);"><strong style="box-sizing:border-box; font-weight:bold"><span style="box-sizing:border-box">Sebastian Smith<br style="box-sizing:border-box">
</span></strong></span><span style="box-sizing:border-box"><span style="box-sizing:border-box">High-Performance Computing Engineer<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">Office of Information Technology<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">1664 North Virginia Street<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box">MS 0291<br style="box-sizing:border-box">
</span><span style="box-sizing:border-box"></span><span style="box-sizing:border-box"></span><br style="box-sizing:border-box">
<strong style="box-sizing:border-box; font-weight:bold">work-phone:<span> </span></strong><span style="box-sizing:border-box"><a href="tel:7756825050" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">775-682-5050</a><br style="box-sizing:border-box">
</span><strong style="box-sizing:border-box; font-weight:bold"></strong><span style="box-sizing:border-box"></span><strong style="box-sizing:border-box; font-weight:bold">email:<span> </span></strong><span style="box-sizing:border-box"><a href="mailto:stsmith@unr.edu" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">stsmith@unr.edu</a><br style="box-sizing:border-box">
</span><strong style="box-sizing:border-box; font-weight:bold">website:<span> </span></strong><span style="box-sizing:border-box"><a href="http://rc.unr.edu/" style="box-sizing: border-box; background: transparent none repeat scroll 0% 0%; text-decoration: underline;">http://rc.unr.edu</a><br style="box-sizing:border-box">
</span></span><br style="box-sizing:border-box">
</span></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Brian Andrus <toomuchit@gmail.com><br>
<b>Sent:</b> Wednesday, November 4, 2020 10:12 AM<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Using hyperthreaded processors</font>
<div> </div>
</div>
<div>
<p>JC,</p>
<p>One thing you will start finding in HPC is that, by it's goal, hyperthreading is usually a poor fit.</p>
<p>If you are properly utilizing your cores, your jobs will actually be slowed by using hyperthreading. They are not 'extra' cores, but a method of swapping a core to a different workload during an idle cycle. The operative word being 'idle'. The goal of HPC
is get resource usage as close to 100% as possible, so there should be no 'idle' cycles.
<br>
</p>
<p>From Intel's own info:</p>
<p><span style="color:rgb(85,85,85); font-family:intel-clear,tahoma,Helvetica,helvetica,Arial,sans-serif; font-size:16px; font-style:normal; font-variant-ligatures:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; background-color:rgb(243,243,243); text-decoration-style:initial; text-decoration-color:initial; display:inline!important; float:none">
With CPU Hyper-Threading, a PC can process more information in less time and run more background tasks without disruption.<span>
<br>
</span></span></p>
<p>In HPC, we have no background tasks (excluding daemons and such on the node), so you have little to nothing that can be improved. You end up allocating some of the various Core memory to secondary tasks instead of allowing more of it to be used by your primary
task, leading to more fetches, wasting effort.</p>
<p>This is a VERY simplistic description, but the point is that hyperthreading is not a silver bullet that will improve HPC performance if you are maximizing your resource utilization.</p>
<p>Ok, I will get off my soapbox :)</p>
<p>Brian Andrus</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 11/4/2020 7:30 AM, Jean-Christophe HAESSIG wrote:<br>
</div>
<blockquote type="cite">
<pre class="x_moz-quote-pre">Hi,
I would like to make good use of hyperthreaded processors and I already
skimmed through a quantity of posts and documentation.
It is pretty clear that Slurm likes to allocate processing units up to
the core level, and to be able to allocate threads one has to either :
- not declare Sockets/Core/Threads at all, only CPUS and use
SelectTypeParameters=CR_CPU_Memory
- resort to some trickery to declare threads as cores
However, the two options have shortcomings :
By not declaring S/C/T, options like --ntasks-per-core seem to get
ignored, but some programs would benefit from being able to be isolated
on their cores.
Not declaring S/C/T (e.g. C=C*T;T=1) as they are in reality also seems
to mess up CPU binding since Slurm does not have a real view on the
node's topology.
When S/C/T are properly declared (with SelectTypeParameters=
CR_Core_Memory), I run into other problems since the allocator thinks I
want 2 CPUS even if I asked for --ntasks=1. In combination with --mem-
per-cpu, this leads to twice memory allocated as was asked for (which
resembles bug #3879).
Finally, while the documentation explains in several places that
processing units should be allocated best by core, I miss an
explanation on why.
Any other options/advice ? Feel free to correct me if I didn't get
something right.
Thanks,
J.C. Haessig
</pre>
</blockquote>
</div>
</body>
</html>