<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">srun: MPI types are...</div>
<div class="moz-cite-prefix">srun: none</div>
<div class="moz-cite-prefix">srun: openmpi</div>
<div class="moz-cite-prefix">srun: pmix_v3</div>
<div class="moz-cite-prefix">srun: pmi2</div>
<div class="moz-cite-prefix">srun: pmix</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">but I'm not sure that tells me much
about how I am supposed to be building OpenMPI?<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 3/27/23 14:41, Pritchard Jr., Howard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2917887E-FA29-4ED7-9D70-11532EE5A2EC@lanl.gov">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you run<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun –mpi=list<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">what does slurm report?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">That will help in determining what argument
you want to supply for the –mpi srun option.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span
style="font-size:12.0pt;color:black">From: </span></b><span
style="font-size:12.0pt;color:black">slurm-users
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of
Craig <a class="moz-txt-link-rfc2396E" href="mailto:cfreese@super.org"><cfreese@super.org></a><br>
<b>Reply-To: </b>Slurm User Community List
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Monday, March 27, 2023 at 12:38 PM<br>
<b>To: </b><a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com">"slurm-users@lists.schedmd.com"</a>
<a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] OpenMPI and
Slurm clarification?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Can someone please clarify the "best practices" for building
OpenMPI compatible with Slurm?<br>
<br>
<a
href="https://urldefense.com/v3/__https:/slurm.schedmd.com/mpi_guide.html*open_mpi__;Iw!!Bt8fGhp8LhKGRg!Cb86a2IwxgqfT5fv1_eEByDpAyhly3ZdN6Wwl7Wod9FRPx9HBpvFVojIRgu5oSpti_3jOXhNyvJqEMGs$"
moz-do-not-send="true">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a>
tells me what I _can_ do but I'm unclear as to what I _should_
do. <br>
<br>
I've built OpenMPI 4.1.5 with: --with-pmix
--with-libevent=internal --with-hwloc=internal
--with-slurm. If I run an MPI program on my cluster (slurm
18.08.8) with "srun -N2 foo" it seems to work fine.
(slurm.conf has MpiDefault=pmix).<br>
<br>
If I "srun --mpi=openmpi -N2 foo" it chokes with:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">OPAL_ERROR: Unreachable in file
../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at line
112<br>
-------------------------------------------------------------------------------------------------------------------<br>
This application appears to have been direct launched using
"srun",<br>
but OMPI was not build with SLURM's PMI support and
therefore cannot<br>
execute. There are several options for building PMI support
under<br>
SLURM, depending upon the SLURM version you are using:<br>
<br>
version 16.05 or later: you can use SLURM's PMIx support.
THis<br>
require that you configure and uild SLURM --with-pmix.<br>
.<br>
.<br>
.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
So I guess the question is, what is the "right" way to build
OpenMPI with Slurm. Is the fact that my non-Slurm pmix works
"correct" or am I just getting lucky that the various software
I have just happens to be compatible. If I build OpenMPI am I
supposed to use Slurm's pmix/libevent/hwloc or is that
optional. If it's optional when/why might I choose to do so.
If I need Slurm's versions is there some way to find which
pmix/libevent/hwloc my current Slurm install is using? Note:
my sysadmins are not going to be helpful as they think Slurm
18 and OpenMPI 4.0.2a is adequate for users' needs :^(.
<br>
<br>
I like the idea of _not_ tying my OpenMPI to the installed
Slurm just in case our support people ever decide to upgrade
system software.<br>
<br>
Thanks.<o:p></o:p></p>
<p><o:p> </o:p></p>
</div>
</blockquote>
<p><br>
</p>
</body>
</html>