[slurm-users] slurmd-used libs in an NFS share?

[Paul Brunk]

Hi all!

In short, I'm thinking about housing some slurmd-used libs in an NFS
share, and am curious about the risk such sharedness offers to
job-running slurmds (not concerned about the jobs themselves here).

For our next Slurm deployment (not a rolling upgrade), our Rocky8
nodes will be 'statelite' (xCAT), as they are already in our CentOS7
cluster.  They NFS mount a shared root image, which includes Slurm in
/opt.  A separate NFS server provides user home dirs and the
"/usr/local"-like dir we call "/apps".  The /scratch lives in Lustre.

We're going to use CUDA, and non-distro versions of PMIx and hwloc.
For the sake of the RAM-dwelling OS image size on the nodes, I'd like
for these to live in the /apps NFS share, while keeping Slurm in the
OS image "/opt".  This would make CUDA and PMIx and hwloc unavailable
to node slurmds, in the event that the /apps mount fails and the OS
"/" mount does not.  I won't care if slurmd can't start a job at such
times, since the user apps would be unavailable anyway (and our NHC
checks for that).  But is there some risk to the slurmd parents of the
already-running jobs, if those slurmds need to (re-)access those
libraries while they're unavailable?

I've looked at e.g. Nvidia's DeepOps (puts CUDA in an unshared
/usr/local, replicated on each node), and Dell's Omnia (puts CUDA in
an NFS share), Nathan Rini's Docker-scale-out cluster (puts CUDA,
etc. in an unshared /usr/local, replicated on each node), and OpenHPC
(Slurm is in /usr, hwloc in /opt (NFS-shared)).  I've started
deploying a dev Omnivector (thanks, Mike Hanby!) using LXD, to see
what they do, but haven't finished that.

Thanks.  I've seen a few "I'm starting a Slurm cluster" walkthrough
threads online lately, but haven't seen this particular thing
addressed.  I'm aware it might be a non-issue.

--
Paul Brunk, system administrator
Advanced Computing Resource Center
Enterprise IT Svcs, the University of Georgia

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230323/f69ce597/attachment.htm>