<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your use of the –with-pmix on the open mpi configure line is important. Without any args to this configure option open mpi configure will first check if there’s an external pmix which is newer than the one that is included in the openmpi
release tarball. If it is not, the internal pmix is built.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You can check in the config.log whether the internal PMix or an external one was used.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you want to be extra careful, find the location of the PMIx v3 used to build the SLURM PMIx plugin, and then rebuild your open mpi 4.1.5 with<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">./configure … --with-pmix=path_to_pmix_used_for_slurm_pmix_plugin_build ….<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But you may be okay without doing this. You can check this by running your open mpi job with<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun –mpi=pmix_v3 -N2 foo<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">and see if it behaves as expected.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’m not sure what the “openmpi” result from srun –mpi=list is about.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Craig <cfreese@super.org><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Monday, March 27, 2023 at 12:54 PM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] [EXTERNAL] OpenMPI and Slurm clarification?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">srun: MPI types are...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: none<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: openmpi<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmix_v3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmi2<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">srun: pmix<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">but I'm not sure that tells me much about how I am supposed to be building OpenMPI?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 3/27/23 14:41, Pritchard Jr., Howard wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">If you run<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">srun –mpi=list<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">what does slurm report?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">That will help in determining what argument you want to supply for the –mpi srun option.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users
<a href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of Craig
<a href="mailto:cfreese@super.org"><cfreese@super.org></a><br>
<b>Reply-To: </b>Slurm User Community List <a href="mailto:slurm-users@lists.schedmd.com">
<slurm-users@lists.schedmd.com></a><br>
<b>Date: </b>Monday, March 27, 2023 at 12:38 PM<br>
<b>To: </b><a href="mailto:slurm-users@lists.schedmd.com">"slurm-users@lists.schedmd.com"</a>
<a href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] OpenMPI and Slurm clarification?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Can someone please clarify the "best practices" for building OpenMPI compatible with Slurm?<br>
<br>
<a href="https://urldefense.com/v3/__https:/slurm.schedmd.com/mpi_guide.html*open_mpi__;Iw!!Bt8fGhp8LhKGRg!Cb86a2IwxgqfT5fv1_eEByDpAyhly3ZdN6Wwl7Wod9FRPx9HBpvFVojIRgu5oSpti_3jOXhNyvJqEMGs$">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a> tells me what
I _can_ do but I'm unclear as to what I _should_ do. <br>
<br>
I've built OpenMPI 4.1.5 with: --with-pmix --with-libevent=internal --with-hwloc=internal --with-slurm. If I run an MPI program on my cluster (slurm 18.08.8) with "srun -N2 foo" it seems to work fine. (slurm.conf has MpiDefault=pmix).<br>
<br>
If I "srun --mpi=openmpi -N2 foo" it chokes with:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">OPAL_ERROR: Unreachable in file ../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at line 112<br>
-------------------------------------------------------------------------------------------------------------------<br>
This application appears to have been direct launched using "srun",<br>
but OMPI was not build with SLURM's PMI support and therefore cannot<br>
execute. There are several options for building PMI support under<br>
SLURM, depending upon the SLURM version you are using:<br>
<br>
version 16.05 or later: you can use SLURM's PMIx support. THis<br>
require that you configure and uild SLURM --with-pmix.<br>
.<br>
.<br>
.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
So I guess the question is, what is the "right" way to build OpenMPI with Slurm. Is the fact that my non-Slurm pmix works "correct" or am I just getting lucky that the various software I have just happens to be compatible. If I build OpenMPI am I supposed
to use Slurm's pmix/libevent/hwloc or is that optional. If it's optional when/why might I choose to do so. If I need Slurm's versions is there some way to find which pmix/libevent/hwloc my current Slurm install is using? Note: my sysadmins are not going
to be helpful as they think Slurm 18 and OpenMPI 4.0.2a is adequate for users' needs :^(.
<br>
<br>
I like the idea of _not_ tying my OpenMPI to the installed Slurm just in case our support people ever decide to upgrade system software.<br>
<br>
Thanks.<o:p></o:p></p>
<p> <o:p></o:p></p>
</blockquote>
<p><o:p> </o:p></p>
</div>
</body>
</html>