[slurm-users] [External] Fwd: Slurm MySQL database configuration

[Chad Cropper]

Following up to my previous response:

You could have your keepalived/maxscale/mariadb/slurmdbd setup on 2 servers. We chose to break it out for maximum resiliency of backend resource types. You have to have 2 database instances with their own storage space, and use replication. I do not know of any solution where you can have active/passive mariadb processes using a single data location.

-Chad

On 7/23/20, 11:24 AM, "slurm-users on behalf of mercan" <slurm-users-bounces at lists.schedmd.com on behalf of ahmet.mercan at uhem.itu.edu.tr> wrote:

    CAUTION: External email. Do not click links or open attachments unless you recognize the sender and know the content is safe.

    Hi;

    I think you can use pacemaker cluster for a virtual slurmdb server. A
    virtual slurmdb server which runs both slurmdb and mysql services on the
    active slurmctl server. When the active slurmctl server die, You can try
    to start on the passive one.

    Regards;

    Ahmet M.


    23.07.2020 19:12 tarihinde Michael Robbert yazdı:
    > Peter,
    > I believe that the answer to your database question is that you don't have two MySQL/MariaDB servers running at the same time. The only way that I know of to run MySQL/MariaDB in an active-active setup, which is what you appear to be describing, is with replication. The other setup is to run it in an active-passive setup using something like Pacemaker/Corosync to manage which node is active at any one time.  The slurmctld and slurmdbd have built in mechanisms to do failover internally, but MySQL/MariaDB doesn't work that way.
    >
    > Mike Robbert
    > Colorado School of Mines
    >
    > On 7/23/20, 09:56, "slurm-users on behalf of Peter Mayes" <slurm-users-bounces at lists.schedmd.com on behalf of pmayes at lenovo.com> wrote:
    >
    >      CAUTION: This email originated from outside of the Colorado School of Mines organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
    >
    >
    >      Hi Folks,
    >
    >      Thanks for responses.
    >
    >      I probably didn't make my initial point totally clear, so following up
    >      with clarification.
    >
    >      The NFS server is considered to be sufficiently highly available
    >      ("Designed for 99.9999% availability with redundant hot-swap components,
    >      including controllers and I/O modules, power supplies, cooling modules")
    >      that we had assumed this was sufficient for the Slurm database, without
    >      additional Database HA features, like Galera, DRBD, etc etc
    >
    >      So the idea was to have the two Slurm server nodes, each running
    >      slurmctld, slurmdbd, mariadb, but the two mariadb instances would be
    >      referencing the same physical database, with only one of the Slurm nodes
    >      doing any database updates at any one time.
    >
    >      Is this a possible configuration? If so, how to get both the mariadb
    >      instances to be running simultaneously (with one idle) accessing the
    >      same database files.
    >
    >      It seems silly to have two database instances on the same NFS server
    >      replicating with something like Galera.
    >
    >      Thanks again
    >      --
    >      Peter
    >
    >      -------
    >      Hi,
    >
    >      My first post to the list, so apologies if this is a FAQ,
    >
    >      My configuration has two nodes allocated for Slurm masters, with a
    >      highly-available NFS server mounting a filesystem across the two nodes.
    >
    >      I need advice on the best configuration.
    >
    >      I naively thought of having a single MariaDB database located in the NFS
    >      server, with slurmctld, slurmdbd and mariadb running on both slurm
    >      nodes, with only one slurmdbd/mariadb actively touching the database at
    >      any one time.
    >
    >      slurm.conf
    >      ----------
    >      SlurmctldHost=slurm1(a.b.c.1)
    >      SlurmctldHost=slurm2(a.b.c.2)
    >
    >      AccountingStorageHost=slurm1
    >      AccountingStorageBackupHost=slurm2
    >
    >      slurmdbd.conf
    >      -------------
    >
    >      DbdHost=slurm1
    >      DbdBackupHost=slurm2
    >
    >      StorageHost=localhost
    >
    >      But MariaDB doesn't seem to like (possibly understandably) two mysql
    >      daemons accessing a single database instance from two servers.
    >
    >      What would be the best way to configure this setup?
    >
    >      Thanks in advance
    >
    >      -- Peter



________________________________
*** The information contained in this communication may be confidential, is intended only for the use of the recipient(s) named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please return it to the sender immediately and delete the original message and any copies of it. If you have any questions concerning this message, please contact the sender. ***