<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Generating the *.conf files from parseable/testable sources is an interesting idea. You mention nodes.conf and partitions.conf. I can't find any documentation on those. Are you just creating those files and then including them in slurm.conf?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Rob</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Greg Wickham <greg.wickham@kaust.edu.sa><br>
<b>Sent:</b> Wednesday, January 18, 2023 1:38 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Maintaining slurm config files for test and production clusters</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:Wingdings}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif}
span.x_elementtoproof
{}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:72.0pt 72.0pt 72.0pt 72.0pt}
div.x_WordSection1
{}
ol
{margin-bottom:0cm}
ul
{margin-bottom:0cm}
-->
</style>
<div lang="en-SA" link="blue" vlink="purple" style="word-wrap:break-word">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="padding:7px 2px 7px 2px; background-color:#A6A6A6">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="width:100%; background-color:#EAEAEA; padding:7px 5px 7px 15px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
<div>You don't often get email from greg.wickham@kaust.edu.sa. <a href="https://aka.ms/LearnAboutSenderIdentification">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="width:75px; background-color:#EAEAEA; padding:7px 5px 7px 5px; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif; font-size:12px; font-weight:normal; color:#212121; text-align:left; word-wrap:break-word">
</td>
</tr>
</tbody>
</table>
<div>
<div class="x_WordSection1">
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hi Rob,</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Slurm doesn’t have a “validate” parameter hence one must know ahead of time whether the configuration will work or not.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">In answer to your question – yes – on our site the Slurm configuration is altered outside of a maintenance window.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Depending upon the potential impact of the change, it will either be made silently (no announcement) or users are notified on slack that there maybe a brief outage.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Slurm is quite resilient – if slurmctld is down, launching jobs will not happen and user commands will fail. But all existing jobs will keep running.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Our users are quite tolerant as well – letting them know when a potential change may impact their overall experience of the cluster seems to be appreciated.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">On our site the configuration files are not changed directly, but moreover a template engine is used – our slurm configuration data is in YAML files, which are then validated and processed to
generate the slurm.conf / nodes.conf / partitions.conf / topology.conf</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">This provides some surety that adding / removing nodes etc. won’t result in an inadvertent configuration issue.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">We have three clusters (one production, and two test) – all are managed the same way.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Finally, using configuration templating it’s possible to spin up new clusters quite quickly . . . The longest time is spent picking a new cluster name.</span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt"> -Greg</span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt">On 17/01/2023, 23:42, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:</span></p>
</div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span></p>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span class="x_elementtoproof"><span style="font-size:12.0pt; color:black; background:white">So, you have two equal sized clusters, one for test and one for production? Our test cluster is a small handful of
machines compared to our production.</span></span><span style="font-size:11.0pt"></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span class="x_elementtoproof"><span style="font-size:12.0pt; color:black; background:white">We have a test slurm control node on a test cluster with a test slurmdbd host and test nodes, all named specifically
for test. We don't want a situation where our "test" slurm controller node is named the same as our "prod" slurm controller node, because the possibility of mistake is too great. ("I THOUGHT I was on the test network....")</span></span><span style="font-size:11.0pt"></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span class="x_elementtoproof"><span style="font-size:12.0pt; color:black; background:white">Here's the ultimate question I'm trying to get answered.... Does anyone update their slurm.conf file on production
outside of an outage? If so, how do you KNOW the slurmctld won't barf on some problem in the file you didn't see (even a mistaken character in there would do it)? We're trying to move to a model where we don't have downtimes as often, so I need to determine
a reliable way to continue to add features to slurm without having to wait for the next outage. There's no way I know of to prove the slurm.conf file is good, except by feeding it to slurmctld and crossing my fingers.</span></span><span style="font-size:11.0pt"></span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt"> </span></p>
</div>
<div>
<p class="x_MsoNormal" style="margin-left:36.0pt"><span class="x_elementtoproof"><span style="font-size:12.0pt; color:black; background:white">Rob</span></span><span style="font-size:11.0pt"></span></p>
<p class="x_MsoNormal"><span style="font-size:11.0pt"> </span></p>
<p class="x_MsoNormal"><span lang="EN-US" style="font-size:11.0pt">--</span></p>
</div>
</div>
</div>
</div>
</body>
</html>