<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>The save state location is where slurm stores its current
information about jobs. That location is the live data of the
cluster and is what allows it to survive restarts of the
slurmctld. The slurmdbd is almost live information and is not
used by the slurmctld for current job state. Thus if the data in
the save state location is corrupt or old, slurm will either purge
those jobs not in that location or the slurmctld will fail to
start.</p>
<p><br>
</p>
<p>Given the critical nature of the save state location, we taken
the step of rsyncing it to a different location periodically.
However, be advised once you restart the slurmctld with a specific
save state it will rectify all the running jobs. So you have one
shot to get it right. So if you suspect that something happened
to your save state info, don't restart the slurmd's or slurmctld
until you think you have a good copy, else the slurmctld will look
at the save state you gave it and then look at the jobs it sees on
the slurmd and then rectify the two (namely dropping jobs it
doesn't see in both places).</p>
<p><br>
</p>
<p>In general this is one of the reasons we have opted to not do HA
with slurm but rather just rely on a single box with backups of
the slurm save state. Our save state is on a local drive, which
also helps with speed. Our backup is rsynced to our isilon and
snapshotted. That said if I had to go back to one of those
snapshots, I know for a fact that I would be losing jobs. There
really isn't much for it at that point.</p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 8/28/19 10:49 AM, David Baker wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DB7PR04MB4572B73C6C1145B0A0FBFFFEFEA30@DB7PR04MB4572.eurprd04.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
I apologise that this email is a bit vague, however we are keen
to understand the role of the Slurm "StateSave" location. I can
see the value of the information in this location when, for
example, we are upgrading Slurm and the database is temporarily
down, however as I note above we are keen to gain a much better
understanding of this directory. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
We have two Slurm controller nodes (one of them is a backup
controller), and currently we have put the "StateSave" directory
on one of the global GPFS file stores. In other respects Slurm
operates independently of the GPFS file stores -- apart from the
fact that if GPFS fails jobs will subsequently fail. There was a
GPFS failure when I was away from the university. Once GPFS had
been restored they attempted to start Slurm, however the
StateSave data was out of date. They eventually restarted Slurm,
however lost all the queued jobs and the job sequence counter
restarted at one.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Am I correct in thinking the the information in the StateSave
location relates to the state of (a) jobs currently running on
the cluster and (b) jobs queued? Am I also correct in thinking
that this information is not stored in the slurm database? In
other words if you lose the statesave data or it gets corrupted
then you will lose all running/queued jobs? </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Any advice on the management and location of the statesave
directory in a dual controller system would be appreciated,
please.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
</blockquote>
</body>
</html>