<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Ok, thanks. "Coordinating" with sys
admins is problematic so I guess I'll just continue with the
internal pmix and keep an eye out for problems. <br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">At least I know I'm not doing anything
blatantly stupid.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 3/27/23 20:46, Pritchard Jr., Howard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BA541C9E-AE10-484A-9B1C-E5243B73E23D@lanl.gov">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}span.apple-tab-span
{mso-style-name:apple-tab-span;}span.s1
{mso-style-name:s1;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Its not essential to use the pmix lib used
to build the SLURM pmix plugin but it does reduce likelihood
of problems.<o:p></o:p></p>
<p class="MsoNormal">I don’t know how, but there is some way
that the admin installing SLURM can “name” the available pmix
–mpi options.<o:p></o:p></p>
<p class="MsoNormal">For instance on one of our systems, the
admin has built multiple variants of the pmix plugin:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">MPI plugin types are...</span></p>
<p class="p1"><span class="apple-tab-span"> </span><span
class="s1">cray_shasta</span></p>
<p class="p1"><span class="apple-tab-span"> </span><span
class="s1">none</span></p>
<p class="p1"><span class="apple-tab-span"> </span><span
class="s1">pmi2</span></p>
<p class="p1"><span class="apple-tab-span"> </span><span
class="s1">pmix</span></p>
<p class="p1"><span class="s1">specific pmix plugin versions
available: pmix_v2,pmix_v3,pmix_v314,pmix_v4,pmix_v422</span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This naming convention has helped us with
“decoupling” building of Open MPI from SLURM build, but does
mean some coordination with the sys admins.<o:p></o:p></p>
<p class="MsoNormal">We’re using SLURM 22.05.6<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hope this helps,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of
Craig <a class="moz-txt-link-rfc2396E" href="mailto:cfreese@super.org"><cfreese@super.org></a><br>
<b>Reply-To: </b>Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Monday, March 27, 2023 at 2:01 PM<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">"slurm-users@lists.schedmd.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>Re: [slurm-users] [EXTERNAL] OpenMPI and
Slurm clarification?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">conf.log...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">checking if user requested PMI support<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">result: no<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">checking if user requested internal
PMIx support(yes)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">result: no<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">checking for pmix.h in /usr<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">result: not found<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">checking for pmix.h in /usr/include<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">result: not found<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">WARNING: discovered external PMIx
version is less than internal version 3.x<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">WARNING: using internal PMIx<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So is looks like it used the internal
version (which is what I was aiming for) and that's ok by me
since it seems to be working, but if I'm really supposed to
be using the same one that SLURM used then I'm gonna have to
figure out a way to determine what that was/is.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 3/27/23 15:28, Pritchard Jr., Howard
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">HI Craig,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Your use of the –with-pmix on the open
mpi configure line is important. Without any args to this
configure option open mpi configure will first check if
there’s an external pmix which is newer than the one that is
included in the openmpi release tarball. If it is not, the
internal pmix is built.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">You can check in the config.log whether
the internal PMix or an external one was used.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">If you want to be extra careful, find the
location of the PMIx v3 used to build the SLURM PMIx plugin,
and then rebuild your open mpi 4.1.5 with</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">./configure …
--with-pmix=path_to_pmix_used_for_slurm_pmix_plugin_build ….</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">But you may be okay without doing this.
You can check this by running your open mpi job with</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">srun –mpi=pmix_v3 -N2 foo</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">and see if it behaves as expected.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I’m not sure what the “openmpi” result
from srun –mpi=list is about.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Howard</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
</span><a
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users-bounces@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"> on behalf of Craig
</span><a href="mailto:cfreese@super.org"
moz-do-not-send="true"><span style="font-size:12.0pt"><cfreese@super.org></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Reply-To: </b>Slurm User Community List </span><a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Date: </b>Monday, March 27, 2023 at 12:54 PM<br>
<b>To: </b></span><a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt">"slurm-users@lists.schedmd.com"</span></a><span
style="font-size:12.0pt;color:black">
</span><a href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Subject: </b>Re: [slurm-users] [EXTERNAL] OpenMPI
and Slurm clarification?</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">srun: MPI types are...</p>
</div>
<div>
<p class="MsoNormal">srun: none</p>
</div>
<div>
<p class="MsoNormal">srun: openmpi</p>
</div>
<div>
<p class="MsoNormal">srun: pmix_v3</p>
</div>
<div>
<p class="MsoNormal">srun: pmi2</p>
</div>
<div>
<p class="MsoNormal">srun: pmix</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">but I'm not sure that tells me much
about how I am supposed to be building OpenMPI?</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">On 3/27/23 14:41, Pritchard Jr., Howard
wrote:</p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">HI Craig,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">If you run</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">srun –mpi=list</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">what does slurm report?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">That will help in determining what
argument you want to supply for the –mpi srun option.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Howard</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
</span><a
href="mailto:slurm-users-bounces@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users-bounces@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"> on behalf of
Craig
</span><a href="mailto:cfreese@super.org"
moz-do-not-send="true"><span style="font-size:12.0pt"><cfreese@super.org></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Reply-To: </b>Slurm User Community List </span><a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Date: </b>Monday, March 27, 2023 at 12:38 PM<br>
<b>To: </b></span><a
href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt">"slurm-users@lists.schedmd.com"</span></a><span
style="font-size:12.0pt;color:black">
</span><a href="mailto:slurm-users@lists.schedmd.com"
moz-do-not-send="true"><span style="font-size:12.0pt"><slurm-users@lists.schedmd.com></span></a><span
style="font-size:12.0pt;color:black"><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] OpenMPI and
Slurm clarification?</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Can someone please clarify the "best practices" for
building OpenMPI compatible with Slurm?<br>
<br>
<a
href="https://urldefense.com/v3/__https:/slurm.schedmd.com/mpi_guide.html*open_mpi__;Iw!!Bt8fGhp8LhKGRg!Cb86a2IwxgqfT5fv1_eEByDpAyhly3ZdN6Wwl7Wod9FRPx9HBpvFVojIRgu5oSpti_3jOXhNyvJqEMGs$"
moz-do-not-send="true">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a>
tells me what I _can_ do but I'm unclear as to what I
_should_ do. <br>
<br>
I've built OpenMPI 4.1.5 with: --with-pmix
--with-libevent=internal --with-hwloc=internal
--with-slurm. If I run an MPI program on my cluster
(slurm 18.08.8) with "srun -N2 foo" it seems to work fine.
(slurm.conf has MpiDefault=pmix).<br>
<br>
If I "srun --mpi=openmpi -N2 foo" it chokes with:</p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">OPAL_ERROR: Unreachable in file
../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at
line 112<br>
-------------------------------------------------------------------------------------------------------------------<br>
This application appears to have been direct launched
using "srun",<br>
but OMPI was not build with SLURM's PMI support and
therefore cannot<br>
execute. There are several options for building PMI
support under<br>
SLURM, depending upon the SLURM version you are using:<br>
<br>
version 16.05 or later: you can use SLURM's PMIx
support. THis<br>
require that you configure and uild SLURM --with-pmix.<br>
.<br>
.<br>
.</p>
</blockquote>
<p class="MsoNormal"><br>
So I guess the question is, what is the "right" way to
build OpenMPI with Slurm. Is the fact that my non-Slurm
pmix works "correct" or am I just getting lucky that the
various software I have just happens to be compatible. If
I build OpenMPI am I supposed to use Slurm's
pmix/libevent/hwloc or is that optional. If it's optional
when/why might I choose to do so. If I need Slurm's
versions is there some way to find which
pmix/libevent/hwloc my current Slurm install is using?
Note: my sysadmins are not going to be helpful as they
think Slurm 18 and OpenMPI 4.0.2a is adequate for users'
needs :^(.
<br>
<br>
I like the idea of _not_ tying my OpenMPI to the installed
Slurm just in case our support people ever decide to
upgrade system software.<br>
<br>
Thanks.</p>
<p> </p>
</blockquote>
<p> </p>
</blockquote>
<p><o:p> </o:p></p>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>