[slurm-users] Migrate the slurmdbd service to another server
Ole Holm Nielsen
Ole.H.Nielsen at fysik.dtu.dk
Mon Mar 4 14:32:57 UTC 2019
On 3/4/19 2:26 PM, Loris Bennett wrote:
> Ole Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk> writes:
>> We're one of the many Slurm sites which run the slurmdbd database daemon on the
>> same server as the slurmctld daemon. This works without problems at our site
>> given our modest load, however, SchedMD recommends to run the daemons on
>> separate servers.
>> Contemplating how to upgrade our cluster from Slurm 17.11 to 18.08, I've come to
>> appreciate the advantage of running the daemons on separate servers: One can
>> upgrade slurmdbd to 18.08 while keeping slurmctld at 17.11 (for a while at
>> least). This enables us to upgrade to 18.08 in the recommended order without
>> any interruption to our running jobs and without any cluster downtime.
> Can't one do this even with only one server? We have always run both
> slurmctld and slurmdbd on one machine and have performed all the updates
> without any downtime.
For minor upgrade 17.11.x to 17.11.y there is no issue because the MySQL
database layout is unchanged.
Major upgrades such as 17.11 to 18.08 is potentially more risky, see for
example this list thread "Extreme long db upgrade 16.05.6 -> 17.11.3":
I recommend to study the instructions in
See also the slides on "Upgrading" in
https://slurm.schedmd.com/SLUG18/field_notes2.pdf from the SLUG meeting
Updating the database layout during a Slurm major upgrade can in special
situations lead to problems, so it's safer to do the upgrade separately
for slurmdbd and slurmctld. This is why I've decided to move my
slurmdbd and database to a separate server now. The slurmctld which
governs the entire cluster is thereby unaffected as I "play" with the
database upgrade, and I can upgrade Slurm without any cluster downtime.
I've tested the actual database upgrade on a test server to verify that
it works without problems and to estimate the downtime expected for
slurmdbd. Our database was converted in less than 5 minutes.
I've also purged a lot of old database records for job steps etc. prior
to moving the database, and this reduced our database by a factor of 3.
See my notes in
>> I've been collecting various pieces of information about Slurm upgrades and I've
>> come up with a tested procedure for migrating the slurmdbd service (on a
>> CentOS/RHEL 7 system) to a new server:
>> The basic idea is that slurmctld continues happily while slurmdbd is down, so
>> you can migrate the MySQL database and slurmdbd behind the scenes. When the new
>> slurmdbd server is up and running, you reconfigure slurm.conf on the cluster.
>> Upgrading slurmctld and slurmd is another topic, and this is discussed in my
>> Wiki page https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm.
>> I'd appreciate comments and suggestions about my procedure.
More information about the slurm-users