On 5/16/24 20:27, Yuengling, Philip J. via slurm-users wrote:
I'm writing up some Ansible code to manage Slurm software updates, and I haven't found any documentation about slurmdbd behavior if the mysql/mariadb database doesn't upgrade successfully.
I would discourage the proposed Slurm updates automatically using Ansible or any other automation tool! Unexpected bugs might come to the surface during upgrading!
The mysql/mariadb database service isn't affected by Slurm updates, although the database contents are changed of course :-)
You need to very carefully make a dry-run slurmdbd update on a test node before doing the actual slurmdbd upgrade, and you need to make a backup of the database before upgrading!
Updates of slurmctld must also be made very carefully with a backup of the spool directory (just in case).
The slurmd in most cases can be upgraded with now or small issues.
My Slurm upgrading notes are in this Wiki page: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#upgrading-slur...
What I do know is that if it is sucessful I can expect to see "Conversion done: success!" in the slurmdbd log. This is good, but minor updates do not update the database as far as I know.
If the Slurm database cannot upgrade upon an update, does it always shut down with a fatal error? What other behaviors should I look for if there is a failure?
IHTH, Ole