<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.xelementtoproof
{mso-style-name:x_elementtoproof;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi Rob,<o:p></o:p></span></p>
<p class="MsoNormal" style="background:white"><span style="color:black;mso-fareast-language:EN-US"><br>
> </span><span style="font-size:12.0pt;color:black">Are you just creating those files and then including them in slurm.conf?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><br>
Yes.<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">We’re using puppet, but you could get the same results using jinja2.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><br>
The workflow we use is a little convoluted – the original YAML files are validated then JSON formatted data is written to intermediate files.<br>
<br>
The schema of the JSON formatted files is rather verbose to match the capability of the template engine (this simplifies the template definition significantly)<br>
<br>
When puppet runs it loads the JSON and then renders the Slurm files.<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">(The same YAML files are used to configure warewulf/ dnsmasq (DHCP) / bind / iPXE . . .)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><br>
Finally, it’s worth mentioning that the YAML files are managed by git, with a gitlab runner completing the validation phase before any files are published to production.<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"> -greg<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:36.0pt">
<b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Groner, Rob <rug262@psu.edu><br>
<b>Date: </b>Wednesday, 18 January 2023 at 5:22 pm<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXTERNAL] Re: [slurm-users] Maintaining slurm config files for test and production clusters<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="font-size:12.0pt;color:black">Generating the *.conf files from parseable/testable sources is an interesting idea. You mention nodes.conf and partitions.conf. I can't find any documentation
on those. Are you just creating those files and then including them in slurm.conf?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="font-size:12.0pt;color:black">Rob<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt;background:white"><span style="font-size:12.0pt;color:black"><o:p> </o:p></span></p>
</div>
<div class="MsoNormal" align="center" style="margin-left:36.0pt;text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="color:black">From:</span></b><span style="color:black"> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Greg Wickham <greg.wickham@kaust.edu.sa><br>
<b>Sent:</b> Wednesday, January 18, 2023 1:38 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Maintaining slurm config files for test and production clusters</span>
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
</div>
</div>
<div>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%">
<tbody>
<tr>
<td width="0%" style="width:.72%;background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt">
</td>
<td width="97%" style="width:97.62%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt">
<div>
<p class="MsoNormal" style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121">You don't often get email from greg.wickham@kaust.edu.sa.
<a href="https://urldefense.com/v3/__https:/aka.ms/LearnAboutSenderIdentification__;!!Nmw4Hv0!xpX8C79DMx78EQjCb-HWl7-Kh8vEmqHmei_dLPEMg9yxAFj9MDVaawDu46eNkUEd6CTvMBDSnrDE6SSaKZiA6Q$">
Learn why this is important</a><o:p></o:p></span></p>
</div>
</td>
<td width="1%" style="width:1.66%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt">
</td>
</tr>
</tbody>
</table>
<div>
<div>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt;color:white">Hi Rob,</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">Slurm doesn’t have a “validate” parameter hence one must know ahead of time whether the configuration will work or not.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">In answer to your question – yes – on our site the Slurm configuration is altered outside of a maintenance window.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">Depending upon the potential impact of the change, it will either be made silently (no announcement) or users are notified on slack that there maybe a brief outage.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">Slurm is quite resilient – if slurmctld is down, launching jobs will not happen and user commands will fail. But all existing jobs will keep running.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">Our users are quite tolerant as well – letting them know when a potential change may impact their overall experience of the cluster seems to be appreciated.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">On our site the configuration files are not changed directly, but moreover a template engine is used – our slurm configuration data is in YAML files, which are then
validated and processed to generate the slurm.conf / nodes.conf / partitions.conf / topology.conf</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">This provides some surety that adding / removing nodes etc. won’t result in an inadvertent configuration issue.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">We have three clusters (one production, and two test) – all are managed the same way.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">Finally, using configuration templating it’s possible to spin up new clusters quite quickly . . . The longest time is spent picking a new cluster name.</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt"> -Greg</span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span style="font-size:11.0pt">On 17/01/2023, 23:42, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:</span><o:p></o:p></p>
</div>
<p class="xmsonormal" style="margin-left:72.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">So, you have two equal sized clusters, one for test and one for production? Our test cluster is a small handful of machines
compared to our production.</span></span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">We have a test slurm control node on a test cluster with a test slurmdbd host and test nodes, all named specifically for
test. We don't want a situation where our "test" slurm controller node is named the same as our "prod" slurm controller node, because the possibility of mistake is too great. ("I THOUGHT I was on the test network....")</span></span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">Here's the ultimate question I'm trying to get answered.... Does anyone update their slurm.conf file on production outside
of an outage? If so, how do you KNOW the slurmctld won't barf on some problem in the file you didn't see (even a mistaken character in there would do it)? We're trying to move to a model where we don't have downtimes as often, so I need to determine a reliable
way to continue to add features to slurm without having to wait for the next outage. There's no way I know of to prove the slurm.conf file is good, except by feeding it to slurmctld and crossing my fingers.</span></span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="xmsonormal" style="margin-left:72.0pt"><span class="xelementtoproof"><span style="font-size:12.0pt;color:black;background:white">Rob</span></span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="xmsonormal" style="margin-left:36.0pt"><span lang="EN-US" style="font-size:11.0pt">--</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>