Thanks for all your help. So it seems we can skip the trouble of compiling SLURM over different mariadb versions.
Tianyang Zhang
SJTU Network Information Center
发件人: Sid Young sid.young@gmail.com 发送时间: 2024年10月30日 7:19 收件人: Andrej Sec andrej.sec@savba.sk 抄送: taleintervenor@sjtu.edu.cn; slurm-users@lists.schedmd.com 主题: Re: [slurm-users] Re: 转发: What is the safe upgrade path when upgrade from slurm21.08 and mariadb5.5?
I recently upgraded from 20.11 to 24.05.2, before moving the cluster from CentOS 7.9 to Oracle Linux 8.10
The DB upgrade should be pretty simple, do a mysqldump first, then uninstall the old DB, change the repo's and install the new DB version. It should recognise the DB files on disk and access them. Do another DB backup on the new DB version. then roll through the Slurm upgrades.
I picked the first and last version of each release, and systematically went through each node till it was done. First the slurm controller node, then the compute nodes. To avoid Job loss, drain the nodes or you end up with a situation where the slurmd can't talk to the running slurmstepd and the job(s) gets lost. (Shows as a "Protocol Error").
Ole sent me a link to this guide which mostly worked.
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrade-slurmd...
Sid Young
W: https://off-grid-engineering.com
On Tue, Oct 29, 2024 at 6:33 PM Andrej Sec via slurm-users <slurm-users@lists.schedmd.com mailto:slurm-users@lists.schedmd.com > wrote:
Hi,
we are facing a similar task. We have a Slurm 22.05 / MariaDB 5.5.68 environment and want to upgrade to a newer version. According to the documentation, it’s recommended to upgrade from 22.05 to a maximum of 23.11 in one step. With the MariaDB upgrade, there’s a challenge between 10.1 and 10.2+ due to incompatible changes (https://mariadb.com/kb/en/changes-improvements-in-mariadb-10-2). This upgrade, as I understand from the documentation, requires at least slurm 22.05, where it is automatically handled by the slurmdbd service.
In the test lab, we performed the following tests:
a. Incremental upgrade - according to MariaDB recommendations:
1. Upgrade MariaDB 5.5.68 -> 10.1.48 -> 10.2.44 2. Start the Slurm suite 22.05, checking content after each MariaDB upgrade step. During the 10.1 -> 10.2 upgrade, the slurmdbd service automatically converted the database to the required format. We had enabled general.log in MariaDB, allowing detailed inspection of database changes during conversion. 3. Upgrade slurmdbd to version 23.11 4. Upgrade slurmctld to version 23.11 5. Upgrade slurmd to version 23.11 6. Check the database content and compare tests before and after the upgrade (we used various reports with scontrol, sreport, sacct, sacctmgr for verification).
b. Direct MariaDB upgrade from 5.5.68 to 10.2.44 using the same approach. According to the tests, this resulted in the same state as the incremental approach.
PS: If you proceed with the upgrade, I would appreciate it if you could let us know about any potential challenges you encountered.
Andrej Sec nscc, Bratislava, Slovakia
_____
Od: "hermes via slurm-users" <slurm-users@lists.schedmd.com mailto:slurm-users@lists.schedmd.com > Komu: slurm-users@lists.schedmd.com mailto:slurm-users@lists.schedmd.com Odoslané: pondelok, 28. október 2024 8:48:19 Predmet: [slurm-users] =?eucgb2312_cn?q?=D7=AA=B7=A2=3A_What_is_the_safe_upgrade_path_when_upgrade_from_slurm21=2E08_and_mariadb5=2E5=3F?=
Hi everyone:
We are currently running business on SLURM21.08 and mariadb5.5.
When talking about the upgrade, we need to keep all the users and jobs history data. And we see the official document wrote:
“When upgrading an existing accounting database to MariaDB 10.2.1 or later from an older version of MariaDB or any version of MySQL, ensure you are running slurmdbd 22.05.7 or later. These versions will gracefully handle changes to MariaDB default values that can cause problems for slurmdbd.”
So is this mean we have to firstly build SLURM>22.05 over mariadb5.5, and do the SLURM upgrade. Then upgrade the mariadb to newer version, and rebuild the same version of SLURM over new mariadb-devel?
And is it safe to jump directly from mariadb5.5 to latest version? How can we check whether the slurm have correctly inherited the historical data?
Thanks,
Tianyang Zhang
SJTU Network Information Center