[slurm-users] Migrate the slurmdbd service to another server

[Loris Bennett]

Hi Ole,

Ole Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk> writes:

> We're one of the many Slurm sites which run the slurmdbd database daemon on the
> same server as the slurmctld daemon.  This works without problems at our site
> given our modest load, however, SchedMD recommends to run the daemons on
> separate servers.
>
> Contemplating how to upgrade our cluster from Slurm 17.11 to 18.08, I've come to
> appreciate the advantage of running the daemons on separate servers: One can
> upgrade slurmdbd to 18.08 while keeping slurmctld at 17.11 (for a while at
> least).  This enables us to upgrade to 18.08 in the recommended order without
> any interruption to our running jobs and without any cluster downtime.

Can't one do this even with only one server?  We have always run both
slurmctld and slurmdbd on one machine and have performed all the updates
without any downtime.

> I've been collecting various pieces of information about Slurm upgrades and I've
> come up with a tested procedure for migrating the slurmdbd service (on a
> CentOS/RHEL 7 system) to a new server:
>
> https://wiki.fysik.dtu.dk/niflheim/Slurm_database#migrate-the-slurmdbd-service-to-another-server
>
> The basic idea is that slurmctld continues happily while slurmdbd is down, so
> you can migrate the MySQL database and slurmdbd behind the scenes.  When the new
> slurmdbd server is up and running, you reconfigure slurm.conf on the cluster.
>
> Upgrading slurmctld and slurmd is another topic, and this is discussed in my
> Wiki page https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm.
>
> I'd appreciate comments and suggestions about my procedure.
>
> /Ole

Cheers,

Loris

-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin         Email loris.bennett at fu-berlin.de