The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”.
My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing.
The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements.
https://slurm.schedmd.com/upgrades.htmlhttps://slurm.schedmd.com/upgrades.html#procedure
In the Upgrade Procedure sectionhttps://slurm.schedmd.com/upgrades.html#procedure, there’re a couple questionable things.
1. Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade.
Before considering the upgrade complete, wait for all jobs that were already running to finish. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems.
within a few paragraphshttps://slurm.schedmd.com/upgrades.html#downtime, this message indicates I will need downtime:
Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon
Further in the guide, in SLURMD (COMPUTE NODES)https://slurm.schedmd.com/upgrades.html#slurmd, I read
Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process
This implies, at least, that existing running jobs can stay running.
2. There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly.
NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … Avoid using low-level package managers like rpm or dpkg as they may not properly enforce these dependencies
However, in SLURMDBD (ACCOUNTING)https://slurm.schedmd.com/upgrades.html#slurmdbd, this statement
Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these
indicates I should be using RPM packages.
Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Windowhttps://slurm.schedmd.com/upgrades.html#compatibility_window that I can update like so:
1. Current = 22.05.3 2. 23.11 3. 25.05 4. 26.05
That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace:
1. Current = 22.05.3 2. 22.05.11 3. 23.02.8 4. 23.11.11 5. 24.05.8 6. 24.11.7 7. 25.05.6 8. 25.11.2
I think those warnings are for the overly cautious. Certainly we have never waited for all jobs to exit before upgrading. Out of paranoia we pause all our jobs, but that is not required. Typically you can upgrade between versions without pausing or canceling jobs. That said you will want to look at the release notes and changelog for the version you want to upgrade to in case there is any issue that is flagged that requires more paranoia. Generally minor version upgrades are fine.
The thing I would note though is this phrase "/Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems."/ What really this is noting is that upgrading in quick succession (especially major upgrades) could be problematic. So say you were to go from 22.05.3 -> 23.11 and then immediately go to 25.05, that could cause problems. If you intend to go from your current version to the latest I recommend spacing out the upgrades, or taking a full downtime.
That said I have never done an upgrade over that large a version change so some one with more experience on the list should be able to answer any questions related to that. My gut says though that if I were trying to step to the latest version I would either clear out the existing jobs, or I would do one upgrade per week to give the jobs on the cluster time to adjust to the new version.
-Paul Edmon-
On 1/20/2026 2:42 PM, Gould, Ron (GRC-VBA0)[AEGIS] via slurm-users wrote:
The tl;dr is “This is my first upgrade since inheriting this Cluster, so I’m not sure what can or can’t be running during the upgrades.”.
My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing.
The Cluster is composed of mostly Oracle Linux 8. I’m aware that I can upgrade within the two release compatibility window. I’ve read through the Upgrade guide and I’m unclear if downtime is required. Perhaps I’m unifying downtime requirements across different SLURM services where I should be interpreting that certain services have their own downtime requirements.
https://slurm.schedmd.com/upgrades.html https://slurm.schedmd.com/upgrades.html#procedure
In the Upgrade Procedure section https://slurm.schedmd.com/upgrades.html#procedure, there’re a couple questionable things.
Is downtime required? Does downtime == “all jobs must be halted”? “Downtime”, to me, seems like nothing should be running. This statement indicates that jobs can be running during the upgrade. / / / Before considering the upgrade complete, /*/wait for all jobs that were already running to finish/*/. Any jobs started before the slurmd system was upgraded will be running with the old version of slurmstepd, so starting another upgrade or trying to use new features in the new version may cause problems./
within a few paragraphs https://slurm.schedmd.com/upgrades.html#downtime, this message indicates I will need downtime: / / / Refer to the expected downtime guidance in the following sections for each relevant Slurm daemon/ / / Further in the guide, in SLURMD (COMPUTE NODES) https://slurm.schedmd.com/upgrades.html#slurmd, I read
/Upgrades will not interrupt running jobs as long as SlurmdTimeout is not reached during the process/
This implies, at least, that existing running jobs can stay running.
There’re conflicting suggestions of using “rpm” to install the RPMs I built with “rpmbuild". Should I use “dnf localinstall ./*.rpm”? I’m inferring that dependencies will (not) be handled correctly. / / / NOTE: If RPM/DEB packages are used, all packages present on each system must be upgraded together instead of piecewise. … /*/Avoid using low-level package managers like rpm or dpkg /*/as they may not properly enforce these dependencies/
However, in SLURMDBD (ACCOUNTING) https://slurm.schedmd.com/upgrades.html#slurmdbd, this statement
/Upgrade the slurmdbd daemon binaries, libraries, and its systemd unit file (if used). If using RPM/DEB packages, the package manager will take care of these/
indicates I should be using RPM packages.
Lastly, to get to a current install, I need to step through multiple versions, with the condition that jobs started with a specific major version must finish within the compatibility window. GitLab has a tool where you plug in your current and intended versions and it tells you explicitly which versions are required along the upgrade path. I’d like a similarly explicit tool for SLURM, but I infer from the Compatibility Window https://slurm.schedmd.com/upgrades.html#compatibility_window that I can update like so:
- Current = 22.05.3
- 23.11
- 25.05
- 26.05
That feels like a big leapfrog between versions. I’d like the practice of upgrading. Is there any detriment to upgrading at a slower pace:
- Current = 22.05.3
- 22.05.11
- 23.02.8
- 23.11.11
- 24.05.8
- 24.11.7
- 25.05.6
- 25.11.2
Thank you for that guidance. I am certainly in the "overly cautious" and "paranoid" groups.
I will probably go through the slower upgrade process (1-8 list), with at least a week between them.
And yes, if anyone has experience doing such a vault between versions, please chime in.
Hi Ron,
I also am in the "paranoid" group. And I've always done updates with jobs "live". Depending on the size of your userbase you may want to consider pausing the submission/start of new jobs while you execute the dnf commands (yes, I use them, rather than the "raw" rpm, because I think they are less error prone with e.g. dependencies). Since you are in the same group as myself, you can save a list of running jobs before and after executing the dnf commands, and see if they match. If they do, congratulations, everything went well. If they don't, there is a (tiny) risk that the jobs which completed during that time might miss "something". Examine their logs and/or warn the users as appropriate. To be clear, this tiny risk is about jobs that would complete *on their own* during that timeframe, not that the slurm update will cause healy jobs to crash. What could happen is a race condition between the jobs terminating and the slurm update which might try to update some information in some DB in an inconsistent way. My understanding is that the job itself (e.g. output file) are safe, it's just the slurm records which might get some trouble.
You mention "waiting at least a week" between a subsequent update, but really the key point is this
*Before considering the upgrade complete, **wait for all jobs that were already running to finish**.*
Which means: if you have a 6h wallclock limit, you can wait only 7h. If you have a 2 months wallclock limit you need to wait for a bit more than 2 months. If you don't have wallclock limit.... you may have to wait forever.... Wait! You have the list of jobs because you are paranoid like myself and made one as mentioned above, so you have to wait "only" for all of them to be completed before proceeding, not "forever".
With these precautions, most likely you won't encounter any issue (of course that gets weighted with the size of the cluster: if you have a huge one with hundreds of thousands of users/jobs/nodes, you will see things that have 0.001% chance of happening and that most of us never encounter)
HTH.
On Tue, Jan 20, 2026 at 1:31 PM Ron Gould via slurm-users < slurm-users@lists.schedmd.com> wrote:
Thank you for that guidance. I am certainly in the "overly cautious" and "paranoid" groups.
I will probably go through the slower upgrade process (1-8 list), with at least a week between them.
And yes, if anyone has experience doing such a vault between versions, please chime in.
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thank you for your pointers and sharing your experience.
My user base is likely small compared to other institutions. Currently, I have about 10 users running about 30 jobs, with some started today and the oldest started in September.
Regarding the "waiting a week" between updates, most of the jobs are short lived, with some taking less than a week. Given that I don't have a short WallClock value, I could update to 23.11 before those long jobs would have to be stopped and restarted under the new slurm dæmons. Doing a couple updates would give me ample practice and I can document the entire thing.
My "slurm_acct_db" database, I have daily, weekly, and monthly backups of it. It's under 2 GB if I had to re-import it. I don't suspect the slurmdbd upgrade will take long.
Prior to that DB backup, I have another script that backs up `${StateSaveLocation}` and "/etc/slurm". This is referenced in "https://slurm.schedmd.com/upgrades.html#backups".
Hi Ron,
On 1/20/26 22:53, Ron Gould via slurm-users wrote:
Thank you for your pointers and sharing your experience.
We always upgrade Slurm while the cluster (700 nodes) is running production jobs, and we never had any issues. As Davide said, the chance of errors seems to be very small. Minor version upgrades should be simple to do because Slurm is basically unchanged. Major version upgrades should be done a little more carefully, just to be on the safe side.
I have collected information on Slurm upgrading, database dumps etc. in these Wiki pages:
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrading-slur...
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_database/#backup-and-restore...
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_database/#backup-and-restore...
Please beware of a MariaDB upgrade issue that was resolved in 22.05.7: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_database/#slurm-database-mod...
IHTH, Ole
My user base is likely small compared to other institutions. Currently, I have about 10 users running about 30 jobs, with some started today and the oldest started in September.
Regarding the "waiting a week" between updates, most of the jobs are short lived, with some taking less than a week. Given that I don't have a short WallClock value, I could update to 23.11 before those long jobs would have to be stopped and restarted under the new slurm dæmons. Doing a couple updates would give me ample practice and I can document the entire thing.
My "slurm_acct_db" database, I have daily, weekly, and monthly backups of it. It's under 2 GB if I had to re-import it. I don't suspect the slurmdbd upgrade will take long.
Prior to that DB backup, I have another script that backs up `${StateSaveLocation}` and "/etc/slurm". This is referenced in "https://slurm.schedmd.com/upgrades.html#backups%3E%3E
On 1/20/26 2:42 pm, Gould, Ron (GRC-VBA0)[AEGIS] via slurm-users wrote:
My Cluster is running an old version, 22.05.3. This is my first upgrade since inheriting the Cluster. As such, I’d like to install 22.05.4 because it’s a short jump, and it fixes the bug my users are seeing.
My one comment would be that you would be better off going to the last release in the 22.05.x series, which was 22.05.11 to get various fixes for security issues in the intervening releases in place.
Changelog:
https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-22.05.md
Best of luck, Chris
Thank you for that point. That had occurred to me, but as this is my first upgrade, I just wanted to upgrade to the version that fixes the bug my users see. I need a li'l win for this Cluster :) . When I have that, I'll take a couple more baby steps and do 22.05.11 with a quick turnaround.
-
Christopher Samuel -
Davide DelVento -
Gould, Ron (GRC-VBA0)[AEGIS] -
Ole Holm Nielsen -
Paul Edmon -
Ron Gould