<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#222222;background:white">Hi Everyone,</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">We have a challenge with scheduling jobs in a partition comprised of heterogenous nodes with respect to memory and cores [1]. We further use cores as the
unit of measure for charging users. Currently we implement a crude mechanism of using MaxMemPerCore as a proxy for memory use, to charge for memory use. In the partition in question, we have nodes with 256GB, 384GB and 768GB of RAM. The 384GB and 256GB nodes
have different core counts, but work out close to ~9GB/core; the 768 nodes are roughly ~18GB/core. The default memory request for the partition is set to this same amount, and will remain unchanged. This partition is really for HTC, so the max node limit
is set to 2 and will remain there (the parallel partition is homogenous). <o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">So, if we increase that MaxMemPerCore number, we'll potentially have a lot of nodes with un-schedulable cores (no memory left) and if we leave it where
it is, the extra 384GB in the larger nodes won't ever get used. Of these two, the former is preferable, even though the charge for memory is effectively halved (that's fine, most allocations are monopoly money anyway). We really just want to optimize job placement
for throughput without having to create a separate partition. <o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">What we're concerned about is this: we don't believe the scheduler will be smart about job placement - placing larger memory jobs preferentially on nodes
with more total memory and smaller memory jobs on the smaller memory nodes. To address this, we're thinking of just weighting the smaller memory nodes more, so that jobs get placed there first, and only get bumped to the larger memory nodes when there are
larger memory requests and when the smaller nodes are already full. <o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">We'd also like this scheme to limit backfill of small jobs on the larger nodes. Ideally, if we can get this to work, we'd extend it be getting rid of the
"largemem" (1-3TB nodes) partition and putting those nodes into this single partition (many of our largemem users could easily fit individual jobs <768GB). I have had good results on a small cluster of very heterogenous nodes all in one large partition with
just letting the scheduler handle things, and it worked reasonably well, with the exception of some very large (bordering on --exclusive or explicitly --exclusive) jobs starving because of small job backfill. <o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">Has anyone (everyone?) tried to deal with this? We're going to go ahead and try out this scheme (it seems pretty straightforward), but I wanted to get a
sense of what other installations are doing. <o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">Best,<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">Scott Ruffner<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">University of Virginia Research Computing<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="font-family:"Arial",sans-serif;color:#222222">[1] Our cluster grows sort of organically as we have an unpredictable budget, and can't plan for forklift replacement of partitions (nodes) on regular lifecycle
periods.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Scott Ruffner<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Senior HPC Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">UVa Research Computing<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(434)924-6778(o)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">(434)295-0250(h)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">sruffner@virginia.edu<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>