[slurm-users] upgrading slurm to 20.11

[Paul Edmon]

Database upgrades can also take a while if your database is large.  
Definitely recommend backing up prior to upgrade as well as running 
slurmdbd -Dvvvvv and not the systemd daemon as if the upgrade takes a 
long time it will kill it preemptively due to unresponsiveness which 
will create all sorts of problems.

-Paul Edmon-

On 5/17/22 2:50 PM, Ole Holm Nielsen wrote:
> Hi,
>
> You can upgrade from 19.05 to 20.11 in one step (2 major releases), 
> skipping 20.02.  When that is completed, it is recommended to upgrade 
> again from 20.11 to 21.08.8 in order to get the current major version. 
> The 22.05 will be out very soon, but you may want to wait a couple of 
> minor releases before upgrading to 22.05.
>
> I have collected much detailed information about Slurm upgrades in my 
> Wiki page:
> https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm
>
> It is strongly recommended to make the dry-run test of the database 
> upgrade, just to be sure your database won't cause problems.
>
> /Ole
>
>
>
> On 17-05-2022 18:13, byron wrote:
>> Sorry, I should have been clearer.   I understand that with regards 
>> to slurmd / slurmctld you can skip a major release without impacting 
>> running jobs etc.  My questions was about upgrading slurmdbd and 
>> whether it was necessary to upgrade through the intermediate major 
>> releases (which I know understand is necessary).
>>
>> Thanks
>>
>>
>> On Tue, May 17, 2022 at 4:49 PM Paul Edmon <pedmon at cfa.harvard.edu 
>> <mailto:pedmon at cfa.harvard.edu>> wrote:
>>
>>     The slurm docs say you can do two major releases at a time
>>     (https://slurm.schedmd.com/quickstart_admin.html
>>     <https://slurm.schedmd.com/quickstart_admin.html>):
>>
>>     "Almost every new major release of Slurm (e.g. 20.02.x to 20.11.x)
>>     involves changes to the state files with new data structures, new
>>     options, etc. Slurm permits upgrades to a new major release from the
>>     past two major releases, which happen every nine months (e.g.
>>     20.02.x or 20.11.x to 21.08.x) without loss of jobs or other state
>>     information."
>>
>>     As for old versions of slurm I think at this point you would need to
>>     contact SchedMD.  I'm sure they have past releases they can hand out
>>     if you are bootstrapping to a newer release.
>>
>>     -Paul Edmon-
>>
>>     On 5/17/22 11:42 AM, byron wrote:
>>>     Thanks Brian for the speedy responce.
>>>
>>>     Am I not correct in thinking that if I just go from 19.05 to 20.11
>>>     then there is the advantage that I can upgrade slurmd and
>>>     slurmctld in one go and it won't affect the running jobs since
>>>     upgrading to a new major release from the past two major releases
>>>     doesn't affect the state information.  Or are you saying that  in
>>>     this case (19.05  direct to 21.08) there isn't any impact to
>>>     running jobs either.  Or did you step through all the versions
>>>     when upgrading slurmd and slurmctld also?
>>>
>>>     Also where do I get a copy of 20.2 from if schedMD aren't
>>>     providing it as a download.
>>>
>>>     Thanks
>>>
>>>
>>>
>>>
>>>     On Tue, May 17, 2022 at 4:05 PM Brian Andrus <toomuchit at gmail.com
>>>     <mailto:toomuchit at gmail.com>> wrote:
>>>
>>>         You need to step upgrade through major versions (not minor).
>>>
>>>         So 19.05=>20.x
>>>
>>>         I would highly recommend going to 21.08 while you are at it.
>>>         I just did the same migration (although they started at 18.x)
>>>         with no
>>>         issues. Running jobs were not impacted and users didn't even
>>>         notice.
>>>
>>>         Brian Andrus
>>>
>>>
>>>         On 5/17/2022 7:35 AM, byron wrote:
>>>         > Hi
>>>         >
>>>         > I'm looking at upgrading our install of slurm from 19.05 to
>>>         20.11 in
>>>         > responce to the recenty announced security vulnerabilities.
>>>         >
>>>         > I've been through the documentation / forums and have
>>>         managed to find
>>>         > the answers to most of my questions but am still unclear
>>>         about the
>>>         > following
>>>         >
>>>         >  - In upgrading the slurmdbd from 19.05 to 20.11 do I need
>>>         to go
>>>         > through all the versions (19.05 => 20.2 => 20.11)? From
>>>         reading the
>>>         > forums it look as though it is necesary
>>>         >
>>> https://groups.google.com/g/slurm-users/c/fftVPaHvTzQ/m/YTWo1mRjAwAJ
>>> <https://groups.google.com/g/slurm-users/c/fftVPaHvTzQ/m/YTWo1mRjAwAJ>
>>>         >
>>> https://groups.google.com/g/slurm-users/c/kXtepX8-L7I/m/udwySA3bBQAJ
>>> <https://groups.google.com/g/slurm-users/c/kXtepX8-L7I/m/udwySA3bBQAJ>
>>>         >    However if that is the case it would seem strange that
>>>         SchedMD have
>>>         > removed 20.2 from the downloads page (I understand the
>>>         reason is that
>>>         > it contains the exploit) if it is still required for the
>>>         upgrade.
>>>         >
>>>         > - We are running version 5.5.68 of the MariaDB, the version
>>>         that comes
>>>         > with centos7.9.   I've seen a few references to upgrading
>>>         v5.5 but
>>>         > they were in the context of upgrading from slurm 17 to 
>>> 18.  I'm
>>>         > wondering if its ok to stick with this version since we're
>>>         already on
>>>         > slurm 19.05.
>>>         >
>>>         > Any help much appreciated.
>