<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thanks to both of you for your replies. I did the move this
morning, and it went off without a hitch. It does appear that the
job state directory keeps track of the queue data, because as soon
as I copied those dirs over, I was able to see the queue
information on the new Slurm controller. <br>
</p>
<p>I had done this operation once before, but it was a couple years
ago, so I just wanted to be safe rather than sorry. Thanks for the
help. <br>
</p>
<p>Prentice<br>
</p>
<div class="moz-cite-prefix">On 1/16/21 1:43 PM, Michael Gutteridge
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALUL84teBQ9gpWVwZxjqGxaXZFhpLyqQouqqy6AFg7VmipNnGg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div class="gmail_default" style="font-family:monospace">I'd
confirm that as well. The state directory has all of that
information. We just upgraded from 18.05 to 20.02 on a
different host and while the cluster was quiet (we had a
maintenance reservation in place) there were running jobs
which survived the upgrade.</div>
<div class="gmail_default" style="font-family:monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace">I think
the big thing to watch out for is setting the slurmdtimeout in
your config prior to the update. Might not be necessary
depending on the exact steps you're using, but it's useful
insurance against job loss.</div>
<div class="gmail_default" style="font-family:monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace">HTH</div>
<div class="gmail_default" style="font-family:monospace"><br>
</div>
<div class="gmail_default" style="font-family:monospace"> -
Michael</div>
<div class="gmail_default" style="font-family:monospace"><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Jan 15, 2021 at 7:51
PM Ryan Novosielski <<a href="mailto:novosirj@rutgers.edu"
moz-do-not-send="true">novosirj@rutgers.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
My understanding is job state directory. Theoretically if
you back it up, screw up and lose it, you can restore it and
try again. There’s some mention of this in the upgrade docs
if I’m not mistaken (as they suggest backing it up in case
you mess up during).
<div><br>
<div dir="ltr"><span
style="background-color:rgba(255,255,255,0)">--</span>
<div><span style="background-color:rgba(255,255,255,0)">#BlackLivesMatter<br>
____<br>
|| \\UTGERS,
|---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a
href="mailto:novosirj@rutgers.edu" dir="ltr"
target="_blank" moz-do-not-send="true">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922
(2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research
Computing - MSB C630, Newark<br>
`'</span></div>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jan 15, 2021, at 13:44,
Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov"
target="_blank" moz-do-not-send="true">pbisbal@pppl.gov</a>>
wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr"><span>Slurm users,</span><br>
<span></span><br>
<span>I'm planning on moving slurmctld and slurmdbd to
a new host. I know how to dump the MySQL DB from the
old server and import it to the new slurmdbd host,
and I know how to copy the job state directories to
the new host. I plan on doing this during our next
maintenance window when there are no jobs running on
the cluster.</span><br>
<span></span><br>
<span>However, there will be plenty of jobs in the
queue, so my question is this: What will happen to
jobs in the queue when I do this? Is the queue
information stored in the database or the job state
directories, or a third location? How can I make
sure I don't lose the state of the queue?</span><br>
<span></span><br>
<span>-- </span><br>
<span>Prentice</span><br>
<span></span><br>
<span></span><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Prentice Bisbal
Lead Software Engineer
Research Computing
Princeton Plasma Physics Laboratory
<a class="moz-txt-link-freetext" href="http://www.pppl.gov">http://www.pppl.gov</a></pre>
</body>
</html>