<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I use rpm's for our installs here. I usually pause all the jobs
prior to the upgrade, then I follow the guide here:</p>
<p><br>
</p>
<p><a class="moz-txt-link-freetext" href="https://slurm.schedmd.com/quickstart_admin.html">https://slurm.schedmd.com/quickstart_admin.html</a></p>
<p><br>
</p>
<p>I haven't done the upgrade to 18.08 though yet, and so I haven't
had to contend with the automatic restart that seems to be the
case with the new rpm spec script (we went to 17.11 prior to the
rpm spec reorg). Frankly I wish that they didn't do the automatic
restart as I like to manage that myself.</p>
<p><br>
</p>
<p>As Chris said though you definitely want to do the slurmdbd
upgrade from the commandline. I've had it where when just
restarting the service it times out and the database only gets
partially update. In which case I had to restore from the
mysqldump I had made and tried again. Also highly recommend doing
mysqldumps prior to major version updates.</p>
<p><br>
</p>
<p>-Paul Edmon-<br>
</p>
<br>
<div class="moz-cite-prefix">On 09/25/2018 09:54 AM, Baker D.J.
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:AM6PR04MB464665A871ECD5C55F642C52FE160@AM6PR04MB4646.eurprd04.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;"
dir="ltr">
<div id="divtagdefaultwrapper" style="font-size: 12pt; color:
rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols;" dir="ltr">
<p style="margin-top:0;margin-bottom:0">Thank you for your
comments. <span style="font-size: 12pt;">I could potentially
force the upgrade of the slurm and slurm-slumdbd rpms
using something like:</span></p>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
rpm -Uvh --noscripts --nodeps --force
slurm-18.08.0-1.el7.x86_64.rpm
slurm-slurmdbd-18.08.0-1.el7.x86_64.rpm<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
That will certainly work, however the slurmctld (or in the
case of my test node, the slurmd) will be killed. The logic
is that at v17.02 the slurm rpm provides slurmctld and
slurmd. So upgrading that rpm will destroy/kill the existing
slurmctld or slurmd processes. That is...</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;">
# rpm -q --whatprovides /usr/sbin/slurmctld</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;">
slurm-17.02.8-1.el7.x86_64</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols; font-size: 16px;
margin-top: 0px; margin-bottom: 0px;">
So if I force the upgrade of that rpm then I delete and
kill <font size="2"
face="Calibri,Helvetica,sans-serif,EmojiFont,Apple Color
Emoji,Segoe UI Emoji,NotoColorEmoji,Segoe UI
Symbol,Android Emoji,EmojiSymbols"><span style="font-size:
16px;">/usr/sbin/slurmctld. In the new rpm structure
slurmctld is now provided by its own rpm.</span></font></div>
<br>
I would have thought that someone would have crossed this
bridge, but maybe most admins don't use rpms...</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color:
rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols;" dir="ltr">
<br>
</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color:
rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols;" dir="ltr">
Best regards,</div>
<div id="divtagdefaultwrapper" style="font-size: 12pt; color:
rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif,
EmojiFont, "Apple Color Emoji", "Segoe UI
Emoji", NotoColorEmoji, "Segoe UI Symbol",
"Android Emoji", EmojiSymbols;" dir="ltr">
David<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b> slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf
of Chris Samuel <a class="moz-txt-link-rfc2396E" href="mailto:chris@csamuel.org"><chris@csamuel.org></a><br>
<b>Sent:</b> 25 September 2018 13:00<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> Re: [slurm-users] Upgrading a slurm on a
cluster, 17.02 --> 18.08</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span
style="font-size:11pt;">
<div class="PlainText">On Tuesday, 25 September 2018
9:41:10 PM AEST Baker D. J. wrote:<br>
<br>
> I guess that the only solution is to upgrade
all the slurm at once. That<br>
> means that the slurmctld will be killed (unless
it has been stopped first).<br>
<br>
We don't use RPMs from Slurm [1], but the rpm
command does have a --noscripts <br>
option to (allegedly, I've never used it) suppress
the execution of pre/post <br>
install scripts.<br>
<br>
A big warning would be do not use systemctl to start
the new slurmdbd for the <br>
first time when upgrading!<br>
<br>
Stop the older one first (and then take a database
dump) and then run the new <br>
slurmdbd process with the "-Dvvv" options (inside
screen, just in case) so <br>
that you can watch its progress and systemd won't
decide it's been taking too <br>
long to start and try and kill it part way through
the database upgrades).<br>
<br>
Once that's completed successfully then you can ^C
it and start it up via the <br>
systemctl once more.<br>
<br>
Hope that's useful!<br>
<br>
All the best,<br>
Chris<br>
<br>
[1] - I've always installed into
${shared_local_area}/slurm/${version} and had <br>
a symlink called "latest" that points at the
currently blessed version of <br>
Slurm. Then I stop slurmdbd, upgrade that as above,
then I can do slurmctld <br>
(with partitions marked down, just in case). Once
those are done I can <br>
restart slurmd's around the cluster.<br>
<br>
-- <br>
Chris Samuel : <a
href="https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C35d2a0583f124e84bf0d08d622deab4e%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=uTVuUTGI3fpPZqffe1p5RifQ1%2BG%2FbsrW0ixkCeu%2FxKw%3D&reserved=0"
id="LPlnk546531" class="OWAAutoLink"
previewremoved="true" moz-do-not-send="true">
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.csamuel.org%2F&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7C35d2a0583f124e84bf0d08d622deab4e%7C4a5378f929f44d3ebe89669d03ada9d8%7C1&sdata=uTVuUTGI3fpPZqffe1p5RifQ1%2BG%2FbsrW0ixkCeu%2FxKw%3D&reserved=0</a>
: Melbourne, VIC<br>
<br>
<br>
<br>
<br>
</div>
</span></font></div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>