<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="en-SA" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hi Richard,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">While trying to respond I was looking into the manual pages and while it does appear that slurm can support some kind of high availability(*) it doesn’t seem simple.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
With multiple slurmctld only one can be active at any time as they share state information. It’s not clear how they know about each other, so this may require STONITH(*).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">With slurmdbd, there’s “</span>AccountingStorageHost<span lang="EN-US">” and “</span>AccountingStorageBackupHost<span lang="EN-US">”, again it’s not quite clear how these interact.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">In slrmdbd.conf there is “</span>StorageBackupHost<span lang="EN-US">” with the description:<br>
<br>
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">. . . </span>. It is up to the backup solution to enforce the coherency of the<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">accounting information between the two hosts. With clustered<span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt">database solutions (active/passive HA), you would not need to use<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">this feature. Default is none.<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"><br>
On our site we’re running only a simple setup. One VM with slurmctld and another VM with both slurmdbd+mariadbd.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Perhaps others who have dabbled with redundancy can reply.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> -greg<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(* I say this trusting the best way to get a response on the Internet is say something wrong and then wait for the avalanche of corrections).</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On 01/11/2022, 12:08, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
<p style="margin-left:36.0pt">Hello Greg,<o:p></o:p></p>
<p style="margin-left:36.0pt">I have a two node set up. node1 is primary slurmctld + backup slurmdbd and node2 is primary slurmdbd + backup slurmctld and mysql database host.<o:p></o:p></p>
<p style="margin-left:36.0pt"> My concern is if node 2 goes down, then the backup slurmdbd will take over, then what will happen ?<o:p></o:p></p>
<p style="margin-left:36.0pt">I have read that slurmctld can cache data, but what about slurmdbd? Not sure.<o:p></o:p></p>
<p style="margin-left:36.0pt">I have intentionally used the slurmdbd + mariadb in the second node because I didn't want to overload the primary slurmctld.<o:p></o:p></p>
<p style="margin-left:36.0pt">I hope you all are getting the picture of how my set up is.<o:p></o:p></p>
<p style="margin-left:36.0pt">Thanks,<o:p></o:p></p>
<p style="margin-left:36.0pt">RC<o:p></o:p></p>
<p style="margin-left:36.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt">On 11/1/2022 10:40 AM, Greg Wickham wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">Hi Richard,</span><span style="font-size:10.0pt"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">Slurmctld caches the updates until slurmdbd comes back online.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">You can see how many records are pending for the database by using the “sdiag” command and looking for “DBD Agent queue size”.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US">If this number grows significantly it means that slurmdbd isn’t available.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"><span lang="EN-US"> -Greg</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:72.0pt">On 01/11/2022, 07:23, "slurm-users"
<a href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-left:72.0pt"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:72.0pt">
Hi,<br>
<br>
Just for my info, I would like to know what happens when SlurmDBD loses <br>
connection to the backend Database, for ex, MariaDB.<br>
<br>
Does it cache the accounting info and keep them till the DB comes back <br>
up ?, or does it panic and shut down ?<br>
<br>
Thank you,<br>
<br>
RC.<br>
<br>
<br>
<o:p></o:p></p>
</div>
</blockquote>
</div>
</body>
</html>