[slurm-users] Ideal NFS exported StateSaveLocation size.

[Paul Edmon]

HA for slurmctld is not multidatacenter HA but rather a traditional HA 
setup where you have two server heads off of one storage brick 
(connected by SAS cables or other fast interconnect).  Multidatacenter 
HA has issues with keeping things in sync due to latency and IOPs (as 
noted below).

So the HA setup for slurmctld will protect you from the server hosting 
the slurmctld getting hosed, not the entire rack going down or the 
datacenter going down.

-Paul Edmon-

On 10/24/2022 4:14 AM, Ole Holm Nielsen wrote:
> On 10/24/22 09:57, Diego Zuccato wrote:
>> Il 24/10/2022 09:32, Ole Holm Nielsen ha scritto:
>>
>>  > It is definitely a BAD idea to store Slurm StateSaveLocation on a 
>> slow
>>  > NFS directory!  SchedMD recommends to use local NVME or SSD disks
>>  > because there will be many IOPS to this file system!
>>
>> IIUC it does have to be shared between controllers, right?
>>
>> Possibly use NVME-backed (or even better NVDIMM-backed) NFS share. Or 
>> replica-3 Gluster volume with NVDIMMs for the bricks, for the 
>> paranoid  :)
>
> IOPS is the key parameter!  Local NVME or SSD should beat any 
> networked storage.  The original question refers to having 
> StateSaveLocation on a standard (slow) NFS drive, AFAICT.
>
> I don't know how many people prefer using 2 slurmctld hosts (primary 
> and backup)?  I certainly don't do that.  Slurm does have a 
> configurable SlurmctldTimeout parameter so that you can reboot the 
> server quickly when needed.
>
> It would be nice if people with experience in HA storage for slurmctld 
> could comment.
>
> /Ole
>