[slurm-users] Migrate the slurmdbd service to another server

[Ole Holm Nielsen]

On 3/4/19 2:26 PM, Loris Bennett wrote:
> Ole Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk> writes:
>> We're one of the many Slurm sites which run the slurmdbd database daemon on the
>> same server as the slurmctld daemon.  This works without problems at our site
>> given our modest load, however, SchedMD recommends to run the daemons on
>> separate servers.
>>
>> Contemplating how to upgrade our cluster from Slurm 17.11 to 18.08, I've come to
>> appreciate the advantage of running the daemons on separate servers: One can
>> upgrade slurmdbd to 18.08 while keeping slurmctld at 17.11 (for a while at
>> least).  This enables us to upgrade to 18.08 in the recommended order without
>> any interruption to our running jobs and without any cluster downtime.
> 
> Can't one do this even with only one server?  We have always run both
> slurmctld and slurmdbd on one machine and have performed all the updates
> without any downtime.

For minor upgrade 17.11.x to 17.11.y there is no issue because the MySQL 
database layout is unchanged.

Major upgrades such as 17.11 to 18.08 is potentially more risky, see for 
example this list thread "Extreme long db upgrade 16.05.6 -> 17.11.3":
https://lists.schedmd.com/pipermail/slurm-users/2018-February/000612.html

I recommend to study the instructions in 
https://slurm.schedmd.com/quickstart_admin.html#upgrade.

See also the slides on "Upgrading" in 
https://slurm.schedmd.com/SLUG18/field_notes2.pdf from the SLUG meeting 
2018 (https://slurm.schedmd.com/publications.html).

Updating the database layout during a Slurm major upgrade can in special 
situations lead to problems, so it's safer to do the upgrade separately 
for slurmdbd and slurmctld.  This is why I've decided to move my 
slurmdbd and database to a separate server now.  The slurmctld which 
governs the entire cluster is thereby unaffected as I "play" with the 
database upgrade, and I can upgrade Slurm without any cluster downtime.

I've tested the actual database upgrade on a test server to verify that 
it works without problems and to estimate the downtime expected for 
slurmdbd.  Our database was converted in less than 5 minutes.

I've also purged a lot of old database records for job steps etc. prior 
to moving the database, and this reduced our database by a factor of 3. 
See my notes in 
https://wiki.fysik.dtu.dk/niflheim/Slurm_database#setting-database-purge-parameters

/Ole

>> I've been collecting various pieces of information about Slurm upgrades and I've
>> come up with a tested procedure for migrating the slurmdbd service (on a
>> CentOS/RHEL 7 system) to a new server:
>>
>> https://wiki.fysik.dtu.dk/niflheim/Slurm_database#migrate-the-slurmdbd-service-to-another-server
>>
>> The basic idea is that slurmctld continues happily while slurmdbd is down, so
>> you can migrate the MySQL database and slurmdbd behind the scenes.  When the new
>> slurmdbd server is up and running, you reconfigure slurm.conf on the cluster.
>>
>> Upgrading slurmctld and slurmd is another topic, and this is discussed in my
>> Wiki page https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm.
>>
>> I'd appreciate comments and suggestions about my procedure.