<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
I started off writing there’s really no particular process for these/just do your changes and start the new software (be mindful of any PATH that might contain data that’s under your software tree, if you have that setup), and that you might need to watch the
timeouts, but I figured I’d have a look at the upgrade guide to be sure.
<div><br>
</div>
<div>There’s really nothing onerous in there. I’d personally back up my database and state save directories just because I’d rather be safe than sorry, or for if have to go backwards and want to be sure. You can run SlurmCtld for a good while with no database
(note that -M on the command line will be broken during that time), just being mindful of the RAM on the SlurmCtld machine/don’t restart it before the DB is back up, and backing up our fairly large database doesn’t take all that long. Whether or not 5 is required
mostly depends on how long you think it will take you to do 6-11 (which could really take you seconds if your process is really as simple as stop, change symlink, start), 12 you’re going to do no matter what, 13 you don’t need if you skipped 5, and 14 is up
to you. So practically, that’s what you’re going to do anyway.</div>
<div><br>
</div>
<div>We just did an upgrade last week, and the only difference is that our compute nodes are stateless, so the compute node upgrades were a reboot (we could upgrade them running, but we did it during a maintenance period anyway, so why?).</div>
<div><br>
</div>
<div>If you want to do this with running jobs, I’d definitely back up the state save directory, but as long as you watch the timeouts, it’s pretty uneventful. You won’t have that long database upgrade period, since no database modifications will be required,
so it’s pretty much like upgrading anything else.</div>
<div><br>
<div>
<div>
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
--<br>
#BlackLivesMatter</div>
<div style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State<span class="Apple-tab-span" style="white-space: pre;"> </span> | Ryan Novosielski - novosirj@rutgers.edu<br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="Apple-tab-span" style="white-space: pre;"> </span> | Office of Advanced Research Computing - MSB A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 28, 2023, at 11:58, Groner, Rob <rug262@psu.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br class="Apple-interchange-newline">
There's 14 steps to upgrading slurm listed on their website, including shutting down and backing up the database. So far we've only updated slurm during a downtime, and it's been a major version change, so we've taken all the steps indicated.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
We now want to upgrade from 23.02.4 to 23.02.5.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Our slurm builds end up in version named directories, and we tell production which one to use via symlink. Changing the symlink will automatically change it on our slurm controller node and all slurmd nodes.</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Is there an expedited, simple, slimmed down upgrade path to follow if we're looking at just a . level upgrade?</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
<br>
</div>
<div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">
Rob</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>