[slurm-users] Ideal NFS exported StateSaveLocation size.
pedmon at cfa.harvard.edu
Mon Oct 24 13:37:21 UTC 2022
HA for slurmctld is not multidatacenter HA but rather a traditional HA
setup where you have two server heads off of one storage brick
(connected by SAS cables or other fast interconnect). Multidatacenter
HA has issues with keeping things in sync due to latency and IOPs (as
So the HA setup for slurmctld will protect you from the server hosting
the slurmctld getting hosed, not the entire rack going down or the
datacenter going down.
On 10/24/2022 4:14 AM, Ole Holm Nielsen wrote:
> On 10/24/22 09:57, Diego Zuccato wrote:
>> Il 24/10/2022 09:32, Ole Holm Nielsen ha scritto:
>> > It is definitely a BAD idea to store Slurm StateSaveLocation on a
>> > NFS directory! SchedMD recommends to use local NVME or SSD disks
>> > because there will be many IOPS to this file system!
>> IIUC it does have to be shared between controllers, right?
>> Possibly use NVME-backed (or even better NVDIMM-backed) NFS share. Or
>> replica-3 Gluster volume with NVDIMMs for the bricks, for the
>> paranoid :)
> IOPS is the key parameter! Local NVME or SSD should beat any
> networked storage. The original question refers to having
> StateSaveLocation on a standard (slow) NFS drive, AFAICT.
> I don't know how many people prefer using 2 slurmctld hosts (primary
> and backup)? I certainly don't do that. Slurm does have a
> configurable SlurmctldTimeout parameter so that you can reboot the
> server quickly when needed.
> It would be nice if people with experience in HA storage for slurmctld
> could comment.
More information about the slurm-users