<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:343213994;
mso-list-template-ids:-2083648116;}
@list l1
{mso-list-id:1510099476;
mso-list-template-ids:1415593990;}
@list l1:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Interesting, I had not read the Licenses feature docs but I will look through that, thanks.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#C00000"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">Raj Sahae | </span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">m. +1 (408) 230-8531</span><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Paul Edmon <pedmon@cfa.harvard.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Friday, July 10, 2020 at 10:09 AM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] How to queue jobs based on non-existent features<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
</div>
<p>Another option would be to use the license feature and just set licenses to 0 when they aren't available.<o:p></o:p></p>
<p>-Paul Edmon-<o:p></o:p></p>
<div>
<p class="MsoNormal">On 7/10/2020 12:42 PM, Raj Sahae wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Hi Brian and Paul,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">You both sent me suggestions about using an offline dummy node with all features set. Thanks for your ideas but this won’t work for me as it’s not practical. We want to allow
users to queue for all supported software versions and that easily numbers in the thousands or tens of thousands (every branch, every commit). If I could make this solution work, I would simply set the Available features for all nodes but this feels like it
won’t scale well, or is an improper use of the Feature capability.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Thanks,</span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#C00000"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">Raj Sahae | </span></b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">m. +1 (408) 230-8531</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Raj Sahae
<a href="mailto:rsahae@tesla.com"><rsahae@tesla.com></a><br>
<b>Date: </b>Thursday, July 9, 2020 at 4:15 PM<br>
<b>To: </b><a href="mailto:slurm-users@schedmd.com">"slurm-users@schedmd.com"</a>
<a href="mailto:slurm-users@schedmd.com"><slurm-users@schedmd.com></a><br>
<b>Subject: </b>How to queue jobs based on non-existent features</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Hi all,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">My apologies if this is sent twice. The first time I sent it without my subscription to the list being complete.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I am attempting to use Slurm as a test automation system for its fairly advanced queueing and job control abilities, and also because it scales very well.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">However, since our use case is a bit outside the standard usage of Slurm, we are hitting some issues that don’t appear to have obvious solutions.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">In our current setup, the Slurm nodes are hosts attached to a test system. Our pipeline (greatly simplified) would be to install some software on the test system
and then run sets of tests against it.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">In our old pipeline, this was done in a single job, however with Slurm I was hoping to decouple these two actions as it makes the entire pipeline more robust to update
failures and would give us more finely grained job control for the actual test run.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">I would like to allow users to queue jobs with constraints indicating which software version they need. Then separately some automated job would scan the queue, see
jobs that are not being allocated due to missing resources, and queue software installs appropriately. We attempted to do this using the Active/Available Features configuration. We use HealthCheck and Epilog scripts to scrape the test system for software properties
(version, commit, etc.) and assign them as Features. Once an install is complete and the Features are updated, queued jobs would start to be allocated on those nodes.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Herein lies the conundrum. If a user submits a job, constraining to run on Version A, but all nodes in the cluster are currently configured with Features=Version-B,
Slurm will fail to queue the job, indicating an invalid feature specification. I completely understand why Features are implemented this way, so my question is, is there some workaround or other Slurm capabilities that I could use to achieve this behavior?
Otherwise my options seem to be:</span><o:p></o:p></p>
<ol style="margin-top:0in" start="1" type="1">
<li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo3"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Go back to how we did it before. The pipeline would have the same level of robustness as before but at least we would still be able
to leverage other queueing capabilities of Slurm.</span><o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo3"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Write our own Feature or Job Submit plugin that customizes this behavior just for us. Seems possible but adds lead time and complexity
to the situation.</span><o:p></o:p></li></ol>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">It's not feasible to update the config for all branches/versions/commits to be AvailableFeatures, as our branch ecosystem is quite large and the maintenance of that
approach would not scale well.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><b><span lang="EN-GB" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">Raj Sahae | Manager, Software QA</span></b><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span lang="FR" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">3500 Deer Creek Rd, Palo Alto, CA 94304</span><o:p></o:p></p>
<p class="MsoNormal" style="line-height:14.3pt"><span lang="FR" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F">m. +1 (408) 230-8531 | </span><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"><a href="file:///composeviewinternalloadurl/%3Cmailto:rsahae@tesla.com%3E">rsahae@tesla.com</a></span><o:p></o:p></p>
<p class="MsoNormal"><span lang="FR" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:black"> </span><o:p></o:p></p>
<p class="MsoNormal"><a href="http://www.tesla.com"><span style="color:windowtext;text-decoration:none"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"><img border="0" width="128" height="13" style="width:1.3333in;height:.1354in" id="Picture_x0020_1" src="cid:image001.png@01D656A4.17EB7320"></span></span></a><span lang="FR" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#7F7F7F"> </span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</blockquote>
</div>
</body>
</html>