[slurm-users] Extreme long db upgrade 16.05.6 -> 17.11.3
Kurt H Maier
khm at sciops.net
Wed Feb 21 17:07:25 MST 2018
On Wed, Feb 21, 2018 at 11:56:38PM +0000, Christopher Benjamin Coffey wrote:
> We have been trying to upgrade slurm on our cluster from 16.05.6 to 17.11.3. I'm thinking this should be doable? Past upgrades have been a breeze, and I believe during the last one, the db upgrade took like 25 minutes. Well now, the db upgrade process is taking far too long. We previously attempted the upgrade during a maintenance window and the upgrade process did not complete after 24 hrs. I gave up on the upgrade and reverted the slurm version back by restoring a backup db.
We hit this on our try as well: upgrading from 17.02.9 to 17.11.3. We
truncated our job history for the upgrade, and then did the rest of the
conversion out-of-band and re-imported it after the fact. It took us
almost sixteen hours to convert a 1.5 million-job store.
We got hung up on precisely the same query you did, on a similarly hefty
machine. It caused us to roll back an upgrade and try again during our
subsequent maintenance window with the above approach.
More information about the slurm-users