<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve just started watching “the ABCs of OpenMPI” videos (they and slides are available at
<a href="https://www.open-mpi.org/video/?category=general">https://www.open-mpi.org/video/?category=general</a> ). I’ve not finished part 1 (of 3) yet, and already have seen very-well-explained material specifically about the building of OpenMPI, Slurm, and
related libraries.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">These are a fantastic resource!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Paul Brunk, system administrator</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Advanced Computing Resource Center<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Enterprise IT Svcs, the University of Georgia<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 3/27/23, 2:29 PM, "slurm-users" <slurm-users-bounces@lists.schedmd.com> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">You don't often get email from <a href="mailto:cfreese@super.org">
cfreese@super.org</a>. Learn why this is important <<a href="https://aka.ms/LearnAboutSenderIdentification%3e">https://aka.ms/LearnAboutSenderIdentification></a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[EXTERNAL SENDER - PROCEED CAUTIOUSLY] <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Can someone please clarify the "best practices" for building OpenMPI compatible with Slurm?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://slurm.schedmd.com/mpi_guide.html#open_mpi">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a> <<a href="https://slurm.schedmd.com/mpi_guide.html#open_mpi>">https://slurm.schedmd.com/mpi_guide.html#open_mpi></a> tells
me what I _can_ do but I'm unclear as to what I _should_ do. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I've built OpenMPI 4.1.5 with: --with-pmix --with-libevent=internal --with-hwloc=internal --with-slurm. If I run an MPI program on my cluster (slurm 18.08.8) with "srun -N2 foo" it seems to work fine. (slurm.conf has MpiDefault=pmix).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If I "srun --mpi=openmpi -N2 foo" it chokes with:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">OPAL_ERROR: Unreachable in file ../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at line 112<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-------------------------------------------------------------------------------------------------------------------<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">This application appears to have been direct launched using "srun",<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">but OMPI was not build with SLURM's PMI support and therefore cannot<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">execute. There are several options for building PMI support under<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">SLURM, depending upon the SLURM version you are using:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">version 16.05 or later: you can use SLURM's PMIx support. THis<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">require that you configure and uild SLURM --with-pmix.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So I guess the question is, what is the "right" way to build OpenMPI with Slurm. Is the fact that my non-Slurm pmix works "correct" or am I just getting lucky that the various software I have just happens to be compatible. If I build OpenMPI
am I supposed to use Slurm's pmix/libevent/hwloc or is that optional. If it's optional when/why might I choose to do so. If I need Slurm's versions is there some way to find which pmix/libevent/hwloc my current Slurm install is using? Note: my sysadmins are
not going to be helpful as they think Slurm 18 and OpenMPI 4.0.2a is adequate for users' needs :^(.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I like the idea of _not_ tying my OpenMPI to the installed Slurm just in case our support people ever decide to upgrade system software.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>