[slurm-users] Ideal NFS exported StateSaveLocation size.

[Ole Holm Nielsen]

On 10/24/22 06:12, Richard Chang wrote:
> Is there a thumb rule for the size of the directory that is NFS exported, 
> and to be used as StateSaveLocation.
> 
> I have a two node Slurmctld setup and both will mount an NFS exported 
> directory as the state save location.

It is definitely a BAD idea to store Slurm StateSaveLocation on a slow NFS 
directory!  SchedMD recommends to use local NVME or SSD disks because 
there will be many IOPS to this file system!

I recommend you to read "Field Notes 6: From The Frontlines of Slurm 
Support", Jason Booth, SchedMD available from 
https://slurm.schedmd.com/publications.html.  Read the Hardware pages 
18-20 which recommend:

Fast path to the StateSaveLocation
■ IOPS this filesystem can sustain is a major bottleneck to job throughput
● At least 2 directories and two files created per job
● The corresponding unlink() calls will add to the load

/Ole