<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
My team lead brought that up also, that we could go ahead and change the symlink that EVERYTHING uses, and nothing would happen...until the service is restarted. That's good that it's not a timing-related change. Of course, we do run the risk that a node
will variously reboot on its own, and thus pick up the change before we're expecting it to. For patch level changes, that really wouldn't be a problem, but if we consider doing this for a major version change, then it probably matters more.</div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Ryan Novosielski <novosirj@rutgers.edu><br>
<b>Sent:</b> Friday, September 29, 2023 11:33 AM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Steps to upgrade slurm for a patchlevel change?</font>
<div> </div>
</div>
<div style="line-break:after-white-space">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="border:0; display:table; width:100%; table-layout:fixed; border-collapse:seperate; float:none">
<tbody style="display:block">
<tr>
<td valign="middle" width="1px" cellpadding="7px 2px 7px 2px" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
</td>
<td valign="middle" width="100%" cellpadding="7px 5px 7px 15px" style="width: 100%; padding: 7px 5px 7px 15px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
<div>You don't often get email from novosirj@rutgers.edu. <a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable" id="OWA856ab1d4-ff8e-b266-c7e7-ea591c446c60" class="OWAAutoLink">
Learn why this is important</a></div>
</td>
<td valign="middle" align="left" width="75px" cellpadding="7px 5px 7px 5px" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; overflow-wrap: break-word; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
</td>
</tr>
</tbody>
</table>
<div>I’ll just say, we haven’t done an online/jobs running upgrade recently (in part because we know our database upgrade will take a long time, and we have some processes that rely on -M), but we have done it and it does work fine. So the paranoia isn’t necessary
unless you know that, like us, the DB upgrade time is not tenable (Ole’s wiki has some great suggestions for how to test that, but they aren’t especially Slurm specific, it’s just a dry-run).
<div><br>
</div>
<div>As far as the shared symlink thing goes, I think you’d be fine, dependent on whether or not you have anything else stored in the shared software tree, changing the symlink and just not restarting compute nodes’ slurmd until you’re ready — though again,
you can do this while jobs are running, so there’s not really a reason to wait, except in cases like ours where it’s just easier to reboot the node than one process for running nodes, and then rebooting, and wanting to be sure that the rebooted compute node
and the running upgraded node will operate exactly the same.<br>
<div>
<div></div>
<div><br>
<blockquote type="cite">
<div>On Sep 29, 2023, at 10:10, Paul Edmon <pedmon@cfa.harvard.edu> wrote:</div>
<br class="x_Apple-interchange-newline">
<div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
This is one of the reasons we stick with using RPM's rather than the symlink process. It's just cleaner and avoids the issue of having the install on shared storage that may get overwhelmed with traffic or suffer outages. Also the package manager automatically
removes the previous versions and local installs stuff. I've never been a fan of the symlink method has it runs counter to the entire point and design of Linux and package managers which are supposed to do this heavy lifting for you.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
Rant aside :). Generally for minor upgrades the process is less touchy. For our setup we follow the following process that works well for us, but does create an outage for the period of the upgrade.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
1. Set all partitions to down: This makes sure no new jobs are scheduled.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
2. Suspend all jobs: This makes sure jobs aren't running while we upgrade.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
3. Stop slurmctld and slurmdbd.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
4. Upgrade the slurmdbd. Restart slurmdbd<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
5. Upgrade the slurmd and slurmctld across the cluster.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
6. Restart slurmd and slurmctld simultaneously using choria.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
7. Unsuspend all jobs</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
8. Reopen all partitions.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
For major upgrades we always take a mysqldump and backup the spool for the slurmctld before upgrading just in case something goes wrong. We've had this happen before when the slurmdbd upgrade cut out early (note, always run the slurmdbd and slurmctld upgrades
in -D mode and not via systemctl as systemctl can timeout and kill the upgrade midway for large upgrades).</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
That said I've also skipped steps 1, 2, 7, and 8 before for minor upgrades and it works fine. The slurmd, slurmctld, and slurmdbd can all run on different versions so long as the slurmdbd > slurmctld > slurmd. So if you want to do a live upgrade you can do
it. However out paranoia we general stop everything. The entire process takes about an hour start to finish, with the longest part being the pausing of all the jobs.</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
-Paul Edmon-</div>
<div style="margin-top:0px; margin-bottom:0px; font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div class="x_moz-cite-prefix" style="font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
On 9/29/2023 9:48 AM, Groner, Rob wrote:<br>
</div>
<blockquote type="cite" style="font-family:Menlo-Regular; font-size:11px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none">
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
I did already see the upgrade section of Jason's talk, but it wasn't much about the mechanics of the actual upgrade process, more of a big picture it seemed. It dealt a lot with different parts of slurm at different versions, which is something we don't have.</div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span class="x_ContentPasted1" style="display: inline !important; background-color: rgb(255, 255, 255);"><br>
</span></div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<span class="x_ContentPasted1" style="display: inline !important; background-color: rgb(255, 255, 255);">One little wrinkle here is that while, yes, we're using a symlink to point to what version of slurm is the current one...it's all on a shared filesystem.
So, ALL nodes, slurmdb, slurmctld are using that same symlink. There is no means to upgrade one component at a time. That means to upgrade, EVERYTHING has to come down before it could come back up. Jason's slides seemed to indicate that, if there were separate
symlinks, then I could focus on just the slurmdb first and upgrade it...then focus on slurmctld and upgrade it, and then finally the nodes (take down their slurmd, upgrade the link, bring up slurmd). So maybe that's what I'm missing.</span></div>
<div class="x_elementToProof" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Otherwise, I think what I'm saying is that I see references to a "rolling upgrade", but I don't see any guide to a rolling upgrade. I just see the 14 steps in <a href="https://slurm.schedmd.com/quickstart_admin.html#upgrade" data-auth="Verified" originalsrc="https://slurm.schedmd.com/quickstart_admin.html#upgrade" shash="CeaSxhPplMlytPlMG0Nzt5XIt65tQEWL5FUVIxdCNNvWdODbuPlOIRoTcplpJQc0kKAQa1/adWGQP8HmUUrGCRUw02QU+IMBgwJww1AGDJajQW8b6uSnhoGOnhEmpSgrVnocjqSa5Bd1jLiG+KPjZ51DS5KJ7cE8W4d8YAJxdu4=" id="OWAa7ce5033-eb3d-6319-f608-ba556ef9fbaf" class="x_OWAAutoLink x_moz-txt-link-freetext">https://slurm.schedmd.com/quickstart_admin.html#upgrade</a>,
and I guess I'd always thought of that as the full octane, high fat upgrade. I've only ever done upgrades during one of our many scheduled downtimes, because the upgrades were always to a new major version, and because I'm a scared little chicken, so I figured
there were maybe some smaller subset of steps if only upgrading a patchlevel change. Smaller change, less risk, less precautionary steps...? I'm seeing now that's not the case.</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Thank you all for the suggestions!</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Rob</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof x_ContentPasted0" style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:696.765625px">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="x_Apple-converted-space"> </span>slurm-users<span class="x_Apple-converted-space"> </span><a href="mailto:slurm-users-bounces@lists.schedmd.com" class="x_moz-txt-link-rfc2396E OWAAutoLink" id="OWA111b0f34-fe7d-3216-85dc-1c4bfe01dce9"><slurm-users-bounces@lists.schedmd.com></a><span class="x_Apple-converted-space"> </span>on
behalf of Ryan Novosielski<span class="x_Apple-converted-space"> </span><a href="mailto:novosirj@rutgers.edu" class="x_moz-txt-link-rfc2396E OWAAutoLink" id="OWAdce4fac6-cf13-1ca7-0388-2e52aa573556"><novosirj@rutgers.edu></a><br>
<b>Sent:</b><span class="x_Apple-converted-space"> </span>Friday, September 29, 2023 2:48 AM<br>
<b>To:</b><span class="x_Apple-converted-space"> </span>Slurm User Community List<span class="x_Apple-converted-space"> </span><a href="mailto:slurm-users@lists.schedmd.com" class="x_moz-txt-link-rfc2396E OWAAutoLink" id="OWAd7edc3ec-b06a-4274-2425-a6de7ebadb51"><slurm-users@lists.schedmd.com></a><br>
<b>Subject:</b><span class="x_Apple-converted-space"> </span>Re: [slurm-users] Steps to upgrade slurm for a patchlevel change?</font>
<div> </div>
</div>
<div style="line-break:after-white-space">
<table width="100%" cellspacing="0" cellpadding="0" border="0" align="left" style="border:0px; display:table; width:711px; table-layout:fixed; float:none">
<tbody style="display:block">
<tr>
<td cellpadding="7px 2px 7px 2px" width="1px" valign="middle" style="padding: 7px 2px; background-color: rgb(166, 166, 166);">
<br>
</td>
<td cellpadding="7px 5px 7px 15px" width="100%" valign="middle" style="width: 677px; padding: 7px 5px 7px 15px; text-align: left; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
<div>You don't often get email from<span class="x_Apple-converted-space"> </span><a href="mailto:novosirj@rutgers.edu" class="x_moz-txt-link-abbreviated OWAAutoLink" id="OWA0399cfd3-068b-0c43-c510-cb009260c0fc">novosirj@rutgers.edu</a>.<span class="x_Apple-converted-space"> </span><a href="https://aka.ms/LearnAboutSenderIdentification" data-auth="NotApplicable" id="OWA672df20d-4e3b-68a8-d62c-23c3360e54a6" class="x_OWAAutoLink">Learn
why this is important</a></div>
</td>
<td cellpadding="7px 5px 7px 5px" width="75px" valign="middle" align="left" style="width: 75px; padding: 7px 5px; font-family: wf_segoe-ui_normal, "Segoe UI", "Segoe WP", Tahoma, Arial, sans-serif; font-size: 12px; font-weight: normal; text-align: left; color: rgb(33, 33, 33); background-color: rgb(234, 234, 234);">
<br>
</td>
</tr>
</tbody>
</table>
<div>I started off writing there’s really no particular process for these/just do your changes and start the new software (be mindful of any PATH that might contain data that’s under your software tree, if you have that setup), and that you might need to watch
the timeouts, but I figured I’d have a look at the upgrade guide to be sure.
<div><br>
</div>
<div>There’s really nothing onerous in there. I’d personally back up my database and state save directories just because I’d rather be safe than sorry, or for if have to go backwards and want to be sure. You can run SlurmCtld for a good while with no database
(note that -M on the command line will be broken during that time), just being mindful of the RAM on the SlurmCtld machine/don’t restart it before the DB is back up, and backing up our fairly large database doesn’t take all that long. Whether or not 5 is required
mostly depends on how long you think it will take you to do 6-11 (which could really take you seconds if your process is really as simple as stop, change symlink, start), 12 you’re going to do no matter what, 13 you don’t need if you skipped 5, and 14 is up
to you. So practically, that’s what you’re going to do anyway.</div>
<div><br>
</div>
<div>We just did an upgrade last week, and the only difference is that our compute nodes are stateless, so the compute node upgrades were a reboot (we could upgrade them running, but we did it during a maintenance period anyway, so why?).</div>
<div><br>
</div>
<div>If you want to do this with running jobs, I’d definitely back up the state save directory, but as long as you watch the timeouts, it’s pretty uneventful. You won’t have that long database upgrade period, since no database modifications will be required,
so it’s pretty much like upgrading anything else.</div>
<div><br>
<div>
<div>
<div dir="auto" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="auto" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="auto" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div dir="auto" style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<div style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
--<br>
#BlackLivesMatter</div>
<div style="letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State<span class="x_x_Apple-tab-span" style="white-space:pre"> </span> | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" class="x_moz-txt-link-abbreviated OWAAutoLink" id="OWAdf0fd8ed-b729-2d4f-5dcf-ce2e8130189f">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ<span class="x_x_Apple-tab-span" style="white-space:pre"> </span> | Office of Advanced Research Computing - MSB A555B, Newark<br>
`'</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
<blockquote type="cite">
<div>On Sep 28, 2023, at 11:58, Groner, Rob<span class="x_Apple-converted-space"> </span><a href="mailto:rug262@psu.edu" class="x_moz-txt-link-rfc2396E OWAAutoLink" id="OWAb17da6f2-4a44-2375-1776-f95333aae831"><rug262@psu.edu></a><span class="x_Apple-converted-space"> </span>wrote:</div>
<br class="x_x_Apple-interchange-newline">
<div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br class="x_x_Apple-interchange-newline">
There's 14 steps to upgrading slurm listed on their website, including shutting down and backing up the database. So far we've only updated slurm during a downtime, and it's been a major version change, so we've taken all the steps indicated.</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
We now want to upgrade from 23.02.4 to 23.02.5.</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Our slurm builds end up in version named directories, and we tell production which one to use via symlink. Changing the symlink will automatically change it on our slurm controller node and all slurmd nodes.</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Is there an expedited, simple, slimmed down upgrade path to follow if we're looking at just a . level upgrade?</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Rob</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</body>
</html>