<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">HI Craig,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">If you run<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">srun –mpi=list<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">what does slurm report?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">That will help in determining what argument you want to supply for the –mpi srun option.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Craig <cfreese@super.org><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Monday, March 27, 2023 at 12:38 PM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] OpenMPI and Slurm clarification?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Can someone please clarify the "best practices" for building OpenMPI compatible with Slurm?<br>
<br>
<a href="https://urldefense.com/v3/__https:/slurm.schedmd.com/mpi_guide.html*open_mpi__;Iw!!Bt8fGhp8LhKGRg!Cb86a2IwxgqfT5fv1_eEByDpAyhly3ZdN6Wwl7Wod9FRPx9HBpvFVojIRgu5oSpti_3jOXhNyvJqEMGs$">https://slurm.schedmd.com/mpi_guide.html#open_mpi</a> tells me what
I _can_ do but I'm unclear as to what I _should_ do. <br>
<br>
I've built OpenMPI 4.1.5 with: --with-pmix --with-libevent=internal --with-hwloc=internal --with-slurm. If I run an MPI program on my cluster (slurm 18.08.8) with "srun -N2 foo" it seems to work fine. (slurm.conf has MpiDefault=pmix).<br>
<br>
If I "srun --mpi=openmpi -N2 foo" it chokes with:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">OPAL_ERROR: Unreachable in file ../../../../../opal/mca/pmix/pmix3/pmix3x_client.c at line 112<br>
-------------------------------------------------------------------------------------------------------------------<br>
This application appears to have been direct launched using "srun",<br>
but OMPI was not build with SLURM's PMI support and therefore cannot<br>
execute. There are several options for building PMI support under<br>
SLURM, depending upon the SLURM version you are using:<br>
<br>
version 16.05 or later: you can use SLURM's PMIx support. THis<br>
require that you configure and uild SLURM --with-pmix.<br>
.<br>
.<br>
.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
So I guess the question is, what is the "right" way to build OpenMPI with Slurm. Is the fact that my non-Slurm pmix works "correct" or am I just getting lucky that the various software I have just happens to be compatible. If I build OpenMPI am I supposed
to use Slurm's pmix/libevent/hwloc or is that optional. If it's optional when/why might I choose to do so. If I need Slurm's versions is there some way to find which pmix/libevent/hwloc my current Slurm install is using? Note: my sysadmins are not going
to be helpful as they think Slurm 18 and OpenMPI 4.0.2a is adequate for users' needs :^(.
<br>
<br>
I like the idea of _not_ tying my OpenMPI to the installed Slurm just in case our support people ever decide to upgrade system software.<br>
<br>
Thanks.<o:p></o:p></p>
<p><o:p> </o:p></p>
</div>
</body>
</html>