<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">conf.log...</div>
<div class="moz-cite-prefix"><br>
</div>
<blockquote>
<div class="moz-cite-prefix">checking if user requested PMI
support</div>
<div class="moz-cite-prefix">result: no</div>
<div class="moz-cite-prefix">checking if user requested internal
PMIx support(yes)</div>
<div class="moz-cite-prefix">result: no</div>
<div class="moz-cite-prefix">checking for pmix.h in /usr</div>
<div class="moz-cite-prefix">result: not found<br>
</div>
<div class="moz-cite-prefix">checking for pmix.h in /usr/include</div>
<div class="moz-cite-prefix">result: not found</div>
<div class="moz-cite-prefix">WARNING: discovered external PMIx
version is less than internal version 3.x</div>
<div class="moz-cite-prefix">WARNING: using internal PMIx</div>
</blockquote>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">So is looks like it used the internal
version (which is what I was aiming for) and that's ok by me since
it seems to be working, but if I'm really supposed to be using the
same one that SLURM used then I'm gonna have to figure out a way
to determine what that was/is.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 3/27/23 15:28, Pritchard Jr., Howard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:E59813B2-C86A-4AB0-A5AD-ECF562975A6D@lanl.gov">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your use of the –with-pmix on the open mpi
configure line is important. Without any args to this
configure option open mpi configure will first check if
there’s an external pmix which is newer than the one that is
included in the openmpi release tarball. If it is not, the
internal pmix is built.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You can check in the config.log whether the
internal PMix or an external one was used.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you want to be extra careful, find the
location of the PMIx v3 used to build the SLURM PMIx plugin,
and then rebuild your open mpi 4.1.5 with<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">./configure …
--with-pmix=path_to_pmix_used_for_slurm_pmix_plugin_build ….<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But you may be okay without doing this.
You can check this by running your open mpi job with<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun –mpi=pmix_v3 -N2 foo<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">and see if it behaves as expected.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m not sure what the “openmpi” result from
srun –mpi=list is about.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of
Craig <a class="moz-txt-link-rfc2396E" href="mailto:cfreese@super.org"><cfreese@super.org></a><br>
<b>Reply-To: </b>Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Monday, March 27, 2023 at 12:54 PM<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">"slurm-users@lists.schedmd.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] [EXTERNAL] OpenMPI and
Slurm clarification?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">srun: MPI types are...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: none<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: openmpi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmix_v3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmi2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmix<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">but I'm not sure that tells me much about
how I am supposed to be building OpenMPI?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 3/27/23 14:41, Pritchard Jr., Howard
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">If you run<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">srun –mpi=list<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">what does slurm report?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">That will help in determining what
argument you want to supply for the –mpi srun option.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
<a href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true"><slurm-users-bounces@lists.schedmd.com></a>
on behalf of Craig
<a href="mailto:cfreese@super.org"
moz-do-not-send="true"><cfreese@super.org></a><br>
<b>Reply-To: </b>Slurm User Community List <a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true">
<slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Monday, March 27, 2023 at 12:38 PM<br>
<b>To: </b><a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true">"slurm-users@lists.schedmd.com"</a>
<a href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] OpenMPI and
Slurm clarification?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Can someone please clarify the "best practices" for building
OpenMPI compatible with Slurm?<br>
<br>
<a
href="https://urldefense.com/v3/__https:/slurm.schedmd.com/mpi_guide.html*open_mpi__;Iw!!Bt8fGhp8LhKGRg!Cb86a2IwxgqfT5fv1_eEByDpAyhly3ZdN6Wwl7Wod9FRPx9HBpvFVojIRgu5oSpti_3jOXhNyvJqEMGs$"
moz-do-not-send="true">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a>
tells me what I _can_ do but I'm unclear as to what I
_should_ do. <br>
<br>
I've built OpenMPI 4.1.5 with: --with-pmix
--with-libevent=internal --with-hwloc=internal
--with-slurm. If I run an MPI program on my cluster (slurm
18.08.8) with "srun -N2 foo" it seems to work fine.
(slurm.conf has MpiDefault=pmix).<br>
<br>
If I "srun --mpi=openmpi -N2 foo" it chokes with:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">OPAL_ERROR: Unreachable in file
../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at line
112<br>
-------------------------------------------------------------------------------------------------------------------<br>
This application appears to have been direct launched
using "srun",<br>
but OMPI was not build with SLURM's PMI support and
therefore cannot<br>
execute. There are several options for building PMI
support under<br>
SLURM, depending upon the SLURM version you are using:<br>
<br>
version 16.05 or later: you can use SLURM's PMIx support.
THis<br>
require that you configure and uild SLURM --with-pmix.<br>
.<br>
.<br>
.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
So I guess the question is, what is the "right" way to build
OpenMPI with Slurm. Is the fact that my non-Slurm pmix
works "correct" or am I just getting lucky that the various
software I have just happens to be compatible. If I build
OpenMPI am I supposed to use Slurm's pmix/libevent/hwloc or
is that optional. If it's optional when/why might I choose
to do so. If I need Slurm's versions is there some way to
find which pmix/libevent/hwloc my current Slurm install is
using? Note: my sysadmins are not going to be helpful as
they think Slurm 18 and OpenMPI 4.0.2a is adequate for
users' needs :^(.
<br>
<br>
I like the idea of _not_ tying my OpenMPI to the installed
Slurm just in case our support people ever decide to upgrade
system software.<br>
<br>
Thanks.<o:p></o:p></p>
<p> <o:p></o:p></p>
</blockquote>
<p><o:p> </o:p></p>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>