<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Franklin Gothic Book";
panose-1:2 11 5 3 2 1 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle22
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1081874506;
mso-list-template-ids:494015278;}
@list l0:level1
{mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1
{mso-list-id:1436512791;
mso-list-template-ids:1263729688;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi Brian,<br>
<br>
>>For monitoring, I use a combination of netdata+prometheus. Data is gathered whenever the nodes are up and stored for history. Yes, when the nodes are powered down, there are empty gaps, but that is interpreted as the node is powered down.<br>
<br>
Ah time-series will cope much better - at the moment our monitoring system (for compute node health at least) is nagios-like, hence the problem. Though there’s potential the entire cluster’s stack may change at some point, so this problem will be more easy
to deal with (with a change of monitoring system for node health).<br>
<br>
>>For the config, I have no access to DNS for configless so I use a symlink to the slurm.conf file a shared filesystem. This works great. Anytime there are changes, a simple 'scontrol reconfigure' brings all running nodes up to speed and any down nodes will
automatically read the latest.<br>
<br>
Yes, currently we use file based and config written to the compute node’s disks themselves via ansible. Perhaps we will consider moving the file to a shared fs.<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">regards<br>
David<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">-------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">David Simpson - Senior Systems Engineer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">ARCCA, Redwood Building,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">King Edward VII Avenue,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">Cardiff, CF10 3NB
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">David Simpson - peiriannydd uwch systemau<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">ARCCA, Adeilad Redwood,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">King Edward VII Avenue,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">Caerdydd, CF10 3NB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><a href="mailto:simpsond4@cardiff.ac.uk"><span style="font-size:9.0pt;color:#0563C1;mso-fareast-language:EN-GB">simpsond4@cardiff.ac.uk</span></a><span style="font-size:9.0pt;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">+44 29208 74657<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Brian Andrus<br>
<b>Sent:</b> 23 February 2022 15:27<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> Re: [slurm-users] monitoring and update regime for Power Saving nodes<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%;background:#FFFADF;border-collapse:collapse;border:none">
<tbody>
<tr>
<td style="border:none;border-left:solid #FFE657 4.5pt;padding:3.75pt 0cm 3.75pt 0cm">
<p class="MsoNormal" align="center" style="text-align:center"><b><span style="font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:black">External email to Cardiff University -
</span></b><span style="font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:black">Take care when replying/opening attachments or links.</span><span style="font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:black;mso-fareast-language:EN-GB"><o:p></o:p></span></p>
</td>
</tr>
<tr>
<td style="border:none;border-left:solid #FFE657 4.5pt;padding:0cm 0cm 3.75pt 0cm">
<p class="MsoNormal" align="center" style="text-align:center"><b><span style="font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:black">Nid ebost mewnol o Brifysgol Caerdydd yw hwn -
</span></b><span style="font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif;color:black">Cymerwch ofal wrth ateb/agor atodiadau neu ddolenni.<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p>David,<o:p></o:p></p>
<p>For monitoring, I use a combination of netdata+prometheus. Data is gathered whenever the nodes are up and stored for history. Yes, when the nodes are powered down, there are empty gaps, but that is interpreted as the node is powered down.<o:p></o:p></p>
<p>For the config, I have no access to DNS for configless so I use a symlink to the slurm.conf file a shared filesystem. This works great. Anytime there are changes, a simple 'scontrol reconfigure' brings all running nodes up to speed and any down nodes will
automatically read the latest.<o:p></o:p></p>
<p>Brian Andrus<o:p></o:p></p>
<div>
<p class="MsoNormal">On 2/23/2022 2:31 AM, David Simpson wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Hi all,<br>
<br>
Interested to know what common approaches were to:<br>
<br>
<br>
<o:p></o:p></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">Monitoring of power saving nodes (e.g. health of the node), when potentially the monitoring system will see it go up and down. Do you limit to BMC only monitoring/health?<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo3">When you want to make changes to slurm.conf (or anything else) to a node which is down due to power saving (during a maintenance/reservation) what is your approach? Do you end up with
2 slurm.confs (one for power saving and one that keeps everything up, to work on during the maintenance)?<o:p></o:p></li></ol>
<p class="MsoNormal"><br>
thanks<br>
David<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">-------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">David Simpson - Senior Systems Engineer</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">ARCCA, Redwood Building,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">King Edward VII Avenue,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">Cardiff, CF10 3NB </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">David Simpson - peiriannydd uwch systemau</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">ARCCA, Adeilad Redwood,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">King Edward VII Avenue,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">Caerdydd, CF10 3NB</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:simpsond4@cardiff.ac.uk"><span style="font-size:9.0pt;color:#0563C1;mso-fareast-language:EN-GB">simpsond4@cardiff.ac.uk</span></a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.0pt;mso-fareast-language:EN-GB">+44 29208 74657</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>