<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-ligatures:standardcontextual;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi all!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In short, I'm thinking about housing some slurmd-used libs in an NFS<o:p></o:p></p>
<p class="MsoNormal">share, and am curious about the risk such sharedness offers to<o:p></o:p></p>
<p class="MsoNormal">job-running slurmds (not concerned about the jobs themselves here).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For our next Slurm deployment (not a rolling upgrade), our Rocky8<o:p></o:p></p>
<p class="MsoNormal">nodes will be 'statelite' (xCAT), as they are already in our CentOS7<o:p></o:p></p>
<p class="MsoNormal">cluster. They NFS mount a shared root image, which includes Slurm in<o:p></o:p></p>
<p class="MsoNormal">/opt. A separate NFS server provides user home dirs and the<o:p></o:p></p>
<p class="MsoNormal">"/usr/local"-like dir we call "/apps". The /scratch lives in Lustre.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We're going to use CUDA, and non-distro versions of PMIx and hwloc.<o:p></o:p></p>
<p class="MsoNormal">For the sake of the RAM-dwelling OS image size on the nodes, I'd like<o:p></o:p></p>
<p class="MsoNormal">for these to live in the /apps NFS share, while keeping Slurm in the<o:p></o:p></p>
<p class="MsoNormal">OS image "/opt". This would make CUDA and PMIx and hwloc unavailable<o:p></o:p></p>
<p class="MsoNormal">to node slurmds, in the event that the /apps mount fails and the OS<o:p></o:p></p>
<p class="MsoNormal">"/" mount does not. I won't care if slurmd can't start a job at such<o:p></o:p></p>
<p class="MsoNormal">times, since the user apps would be unavailable anyway (and our NHC<o:p></o:p></p>
<p class="MsoNormal">checks for that). But is there some risk to the slurmd parents of the<o:p></o:p></p>
<p class="MsoNormal">already-running jobs, if those slurmds need to (re-)access those<o:p></o:p></p>
<p class="MsoNormal">libraries while they're unavailable?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I've looked at e.g. Nvidia's DeepOps (puts CUDA in an unshared<o:p></o:p></p>
<p class="MsoNormal">/usr/local, replicated on each node), and Dell's Omnia (puts CUDA in<o:p></o:p></p>
<p class="MsoNormal">an NFS share), Nathan Rini's Docker-scale-out cluster (puts CUDA,<o:p></o:p></p>
<p class="MsoNormal">etc. in an unshared /usr/local, replicated on each node), and OpenHPC<o:p></o:p></p>
<p class="MsoNormal">(Slurm is in /usr, hwloc in /opt (NFS-shared)). I've started<o:p></o:p></p>
<p class="MsoNormal">deploying a dev Omnivector (thanks, Mike Hanby!) using LXD, to see<o:p></o:p></p>
<p class="MsoNormal">what they do, but haven't finished that.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks. I've seen a few "I'm starting a Slurm cluster" walkthrough<o:p></o:p></p>
<p class="MsoNormal">threads online lately, but haven't seen this particular thing<o:p></o:p></p>
<p class="MsoNormal">addressed. I'm aware it might be a non-issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black;mso-ligatures:none">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-ligatures:none">Paul Brunk, system administrator</span><span style="mso-ligatures:none"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-ligatures:none">Advanced Computing Resource Center<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black;mso-ligatures:none">Enterprise IT Svcs, the University of Georgia<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>