<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Another option would be to use the license feature and just set
licenses to 0 when they aren't available.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 7/10/2020 12:42 PM, Raj Sahae wrote:<br>
</div>
<blockquote type="cite"
cite="mid:F14ED719-0B45-455A-B92B-129AD77E408A@teslamotors.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1510099476;
mso-list-template-ids:1415593990;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1668358807;
mso-list-template-ids:1144161180;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi
Brian and Paul,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif">You
both sent me suggestions about using an offline dummy node
with all features set. Thanks for your ideas but this won’t
work for me as it’s not practical. We want to allow users to
queue for all supported software versions and that easily
numbers in the thousands or tens of thousands (every branch,
every commit). If I could make this solution work, I would
simply set the Available features for all nodes but this
feels like it won’t scale well, or is an improper use of the
Feature capability.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#C00000"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">Raj
Sahae | </span></b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">m.
+1 (408) 230-8531</span><span
style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span
style="color:black">Raj Sahae <a class="moz-txt-link-rfc2396E" href="mailto:rsahae@tesla.com"><rsahae@tesla.com></a><br>
<b>Date: </b>Thursday, July 9, 2020 at 4:15 PM<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@schedmd.com">"slurm-users@schedmd.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@schedmd.com"><slurm-users@schedmd.com></a><br>
<b>Subject: </b>How to queue jobs based on non-existent
features<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Hi
all,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">My
apologies if this is sent twice. The first time I sent it
without my subscription to the list being complete.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I
am attempting to use Slurm as a test automation system for
its fairly advanced queueing and job control abilities, and
also because it scales very well.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">However,
since our use case is a bit outside the standard usage of
Slurm, we are hitting some issues that don’t appear to have
obvious solutions.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">In
our current setup, the Slurm nodes are hosts attached to a
test system. Our pipeline (greatly simplified) would be to
install some software on the test system and then run sets
of tests against it.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">In
our old pipeline, this was done in a single job, however
with Slurm I was hoping to decouple these two actions as it
makes the entire pipeline more robust to update failures and
would give us more finely grained job control for the actual
test run.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I
would like to allow users to queue jobs with constraints
indicating which software version they need. Then separately
some automated job would scan the queue, see jobs that are
not being allocated due to missing resources, and queue
software installs appropriately. We attempted to do this
using the Active/Available Features configuration. We use
HealthCheck and Epilog scripts to scrape the test system for
software properties (version, commit, etc.) and assign them
as Features. Once an install is complete and the Features
are updated, queued jobs would start to be allocated on
those nodes.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Herein
lies the conundrum. If a user submits a job, constraining to
run on Version A, but all nodes in the cluster are currently
configured with Features=Version-B, Slurm will fail to queue
the job, indicating an invalid feature specification. I
completely understand why Features are implemented this way,
so my question is, is there some workaround or other Slurm
capabilities that I could use to achieve this behavior?
Otherwise my options seem to be:</span><o:p></o:p></p>
<ol style="margin-top:0in" type="1" start="1">
<li class="MsoNormal" style="color:black;mso-list:l0 level1
lfo3"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif">Go
back to how we did it before. The pipeline would have the
same level of robustness as before but at least we would
still be able to leverage other queueing capabilities of
Slurm.</span><o:p></o:p></li>
<li class="MsoNormal" style="color:black;mso-list:l0 level1
lfo3"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif">Write
our own Feature or Job Submit plugin that customizes this
behavior just for us. Seems possible but adds lead time
and complexity to the situation.</span><o:p></o:p></li>
</ol>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">It's
not feasible to update the config for all
branches/versions/commits to be AvailableFeatures, as our
branch ecosystem is quite large and the maintenance of that
approach would not scale well.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><b><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"
lang="EN-GB">Raj Sahae | Manager, Software QA</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"
lang="FR">3500 Deer Creek Rd, Palo Alto, CA 94304</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"
lang="FR">m. +1 (408) 230-8531 | </span><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><a
href="file:///composeviewinternalloadurl/%3Cmailto:rsahae@tesla.com%3E"
moz-do-not-send="true">rsahae@tesla.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"
lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal"><a href="http://www.tesla.com/"
moz-do-not-send="true"><span
style="color:windowtext;text-decoration:none"><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><img
style="width:1.3333in;height:.1354in"
id="Picture_x0020_1"
src="cid:part2.4FD52A93.8535EA4E@cfa.harvard.edu"
class="" width="128" height="13" border="0"></span></span></a><span
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"
lang="FR"> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</blockquote>
</body>
</html>