[slurm-users] SlurmDBD losing connection to the backend MariaDB

[Richard Chang]

Does it mean it is best to use a single slurmdbd host in my case?

My primary slurmctld is the backup slurmdbd host, and my worry is if the 
primary slurmdbd host ( which is also the mariadb server) goes down, 
will the backup slurmdbd be able to cache data and wait till the mariadb 
catches up ?

Thanks,

RC

On 11/2/2022 2:00 AM, Brian Andrus wrote:
> Ole,
>
> Fair enough, it is actually slurmctld that does the caching. Technical 
> typo on my part there.
>
> Just trying to let the user know, there is a window that they have to 
> ensure no information is lost during a database outage.
>
> Brian Andrus
>
> On 11/1/2022 1:43 AM, Ole Holm Nielsen wrote:
>> Hi Brian,
>>
>> On 11/1/22 05:28, Brian Andrus wrote:
>>> It caches up to a point. As I understand it, that is about an hour 
>>> (depending on size and how busy the cluster is, as well as available 
>>> memory, etc).
>>
>> Have you found any documentation of slurmdbd caching?  It's 
>> well-known that slurmctld caches information while slurmdbd is down, 
>> see for example page 30 in the talk "Field Notes Mark 2: Random 
>> Musings From Under A New Hat"[1] by Tim Wickberg, SchedMD:
>>
>>> For slurmdbd, the critical element in the failure domain is
>>> MySQL, not slurmdbd. slurmdbd itself is stateless.
>>> ● slurmctld will cache accounting records (up to a limit) if
>>> slurmdbd is unavailable. This can be hours+ to days+
>>> depending on your system without data loss.
>>
>> The statelessness of slurmdbd makes me think that it can't cache any 
>> data.
>>
>> Thanks,
>> Ole
>>
>> [1] https://slurm.schedmd.com/publications.html
>>
>>> On 10/31/2022 9:20 PM, Richard Chang wrote:
>>>> Hi,
>>>>
>>>> Just for my info, I would like to know what happens when SlurmDBD 
>>>> loses connection to the backend Database, for ex, MariaDB.
>>>>
>>>> Does it cache the accounting info and keep them till the DB comes 
>>>> back up ?, or does it panic and shut down ?
>>
>