<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks for your patience and persistence. I’ll find a place to post your experiences to help others as they navigate these waters.<div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 28, 2017, at 8:52 PM, Philip Kovacs <<a href="mailto:pkdevel@yahoo.com" class="">pkdevel@yahoo.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class=""><div id="yui_3_16_0_ym19_1_1511929322573_4846" class=""><span id="yui_3_16_0_ym19_1_1511929322573_5001" class="">I doubled checked and yes, you definitely want the pmix headers and libpmix library installed before you configure slurm.</span></div><div id="yui_3_16_0_ym19_1_1511929322573_4846" class=""><span id="yui_3_16_0_ym19_1_1511929322573_5730" class="">No need to use --with-pmix </span>if pmix is installed in standard system locations. Configure slurm and it will see the pmix </div><div id="yui_3_16_0_ym19_1_1511929322573_4846" class="">installation. After configuring slurm, but before installing it, manually remove the pmix versions of libpmi.so* and libpmi2.so*. </div><div id="yui_3_16_0_ym19_1_1511929322573_4846" class="">Install slurm and use its versions of those libs. Test every mpi variant seen when you run `srun --mpi=list hostname`. </div><div id="yui_3_16_0_ym19_1_1511929322573_4846" class="">You should see pmi2 and pmix in that list and several others. The pmix option will invoke a slurm plugin that is linked </div><div id="yui_3_16_0_ym19_1_1511929322573_4846" dir="ltr" class="">directly to the libpmix.so library. If you favor using the pmix versions of pmi/pmi2, sounds like you'll get better performance</div><div id="yui_3_16_0_ym19_1_1511929322573_4846" dir="ltr" class="">when using pmi/pmi2, but as mentioned, you would want to test every mpi variant listed to make sure everything works.</div> <div class="qtdSeparateBR"><br class=""><br class=""></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;" class=""> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" class=""> <div dir="ltr" class=""><font size="2" face="Arial" class=""> On Tuesday, November 28, 2017 9:57 PM, "<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>" <<a href="mailto:rhc@open-mpi.org" class="">rhc@open-mpi.org</a>> wrote:<br class=""></font></div> <br class=""><br class=""> <div class="y_msg_container"><div id="yiv5912775310" class=""><div class="">My apologies - I guess we hadn’t been tracking it that way. I’ll try to add some clarification. We presented a nice table at the BoF and I just need to find a few minutes to post it.<div class="yiv5912775310"><br clear="none" class="yiv5912775310"></div><div class="yiv5912775310">I believe you do have to build slurm against PMIx so that the pmix plugin is compiled. You then also have to specify --mpi=pmix so slurm knows to use that plugin for this specific job.</div><div class="yiv5912775310"><br clear="none" class="yiv5912775310"></div><div class="yiv5912775310">You actually might be able to use the PMIx backward compatibility, and you might want to do so with slurm 17.11 and above as Mellanox did a nice job of further optimizing launch performance on IB platforms by adding fabric-based collective implementations to the pmix plugin. If you replace the slurm libpmi and libpmi2 with the ones from PMIx, what will happen is that PMI and PMI2 calls will be converted to their PMIx equivalent and passed to the pmix plugin. This lets you take advantage of what Mellanox did.</div><div class="yiv5912775310"><br clear="none" class="yiv5912775310"></div><div class="yiv5912775310">The caveat is that your MPI might ask for some PMI/PMI2 feature that we didn’t implement. We have tested with MPICH as well as OMPI and it was fine - but we cannot give you a blanket guarantee (e.g., I’m pretty sure MVAPICH won’t work). Probably safer to stick with the slurm libs for that reason unless you test to ensure it all works.</div><div class="yiv5912775310"><br clear="none" class="yiv5912775310"></div><div class="yiv5912775310yqt7075417170" id="yiv5912775310yqt65360"><div class="yiv5912775310"><br clear="none" class="yiv5912775310"><div class=""><blockquote class="yiv5912775310" type="cite"><div class="yiv5912775310">On Nov 28, 2017, at 6:42 PM, Paul Edmon <<a rel="nofollow" shape="rect" class="yiv5912775310" ymailto="mailto:pedmon@cfa.harvard.edu" target="_blank" href="mailto:pedmon@cfa.harvard.edu">pedmon@cfa.harvard.edu</a>> wrote:</div><br clear="none" class="yiv5912775310Apple-interchange-newline"><div class="yiv5912775310">
</div></blockquote></div></div></div></div><div class="yiv5912775310yqt7075417170" id="yiv5912775310yqt57129"><div class=""><div class="yiv5912775310"><div class="yiv5912775310">Okay, I didn't see any note on the PMIx 2.1 page about versions
of slurm it was combatible with so I assumed all of them. My
bad. Thanks for the correction and the help. I just naively used
the rpm spec that was packaged with PMIx which does enable the
legacy support. It seems best then to let PMIx handle pmix solely
and let slurm handle the rest. Thanks!</div><div class="yiv5912775310">Am I right in reading that you don't have to build slurm against
PMIx? So it just interoperates with it fine if you just have it
installed and specify pmix as the launch option? That's neat.<br clear="none" class="yiv5912775310">
</div><div class="yiv5912775310">-Paul Edmon-<br clear="none" class="yiv5912775310">
</div>
<br clear="none" class="yiv5912775310">
<div class="yiv5912775310moz-cite-prefix">On 11/28/2017 6:11 PM, Philip Kovacs
wrote:<br clear="none" class="yiv5912775310">
</div>
<blockquote class="yiv5912775310" type="cite">
<div class="yiv5912775310" style="background-color:rgb(255, 255, 255);font-family:'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;font-size:13px;">
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_53348">Actually if
you're set on installing pmix/pmix-devel from the rpms and
then configuring slurm manually,</div>
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_53348">you could just
move the pmix-installed versions of libpmi.so* and libpmi2.so*
to a safe place, configure</div>
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_53348">and install
slurm which will drop in its versions pf those libs and then
either use the slurm versions or move</div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_53348">the
the pmix versions of libpmi and libpmi2 back into place in
/usr/lib64. </div>
<div class="yiv5912775310qtdSeparateBR"><br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310yahoo_quoted" style="display:block;">
<div class="yiv5912775310" style="font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;">
<div class="yiv5912775310" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;">
<div class="yiv5912775310" dir="ltr"><font class="yiv5912775310" size="2" face="Arial"> On Tuesday,
November 28, 2017 5:32 PM, Philip Kovacs
<a rel="nofollow" shape="rect" class="yiv5912775310moz-txt-link-rfc2396E" ymailto="mailto:pkdevel@yahoo.com" target="_blank" href="mailto:pkdevel@yahoo.com"><pkdevel@yahoo.com></a> wrote:<br clear="none" class="yiv5912775310">
</font></div>
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
<div class="yiv5912775310y_msg_container">
<div class="yiv5912775310" id="yiv5912775310">
<div class="yiv5912775310">
<div class="yiv5912775310" style="background-color:rgb(255, 255, 255);font-family:'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;font-size:13px;">
<div class="yiv5912775310" id="yiv5912775310">
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_19392">
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_19391" style="background-color:rgb(255, 255, 255);font-size:13px;">
<div class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_14678">This
issue is that pmi 2.0+ provides a
"backward compatibility" feature,
enabled by default, which installs</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_19393">both
libpmi.so and libpmi2.so in addition to
libpmix.so. The route with the least
friction for you would probably</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_19394">be
to uninstall pmix, then install slurm
normally, letting it install its libpmi
and libpmi2. Next configure and compile</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_20421">a
custom pmix with that backward </span>feature
_disabled_, so it only installs
libpmix.so. Slurm will "see" the pmix
library</div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style="">after you install it and load it
via its plugin when you use --mpi=pmix.
Again, just use the Slurm pmi and pmi2
and </div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style="">install pmix separately with the
backward compatible option disabled.</div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310"><br clear="none" class="yiv5912775310">
</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_20590">There
is a packaging issue there in which two
packages are trying to install their own
versions of the same files. </span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310">That s</span>hould be
brought to attention of the packages.
Meantime you can work around it.</div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310"><br clear="none" class="yiv5912775310">
</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310">For PMIX:</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><span class="yiv5912775310"><br clear="none" class="yiv5912775310">
</span></div>
<div class="yiv5912775310" dir="ltr" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_13882" style=""><font class="yiv5912775310" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_20598" face="Courier New, courier, monaco, monospace, sans-serif">./configure
--disable-pmi-backward-compatibility //
... etc ...</font></div>
<div class="yiv5912775310qtdSeparateBR" id="yiv5912775310yui_3_16_0_ym19_1_1511904995913_19395" style=""><br clear="none" class="yiv5912775310">
</div>
</div>
</div>
</div>
<div class="yiv5912775310">
<div class="yiv5912775310" style="font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px;">
<div class="yiv5912775310" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;">
<div class="yiv5912775310qtdSeparateBR"><br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310yqt7102444885" id="yiv5912775310yqtfd45512">
<div class="yiv5912775310" dir="ltr"><font class="yiv5912775310" size="2" face="Arial">
On Tuesday, November 28, 2017 4:44 PM,
Artem Polyakov
<a rel="nofollow" shape="rect" class="yiv5912775310moz-txt-link-rfc2396E" ymailto="mailto:artpol84@gmail.com" target="_blank" href="mailto:artpol84@gmail.com"><artpol84@gmail.com></a> wrote:<br clear="none" class="yiv5912775310">
</font></div>
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
<div class="yiv5912775310y_msg_container">
<div class="yiv5912775310" id="yiv5912775310">
<div class="yiv5912775310">
<div class="yiv5912775310" dir="ltr">Hello, Paul
<div class="yiv5912775310"><br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310">Please see below.<br clear="none" class="yiv5912775310">
<div class="yiv5912775310gmail_extra"><br clear="none" class="yiv5912775310">
<div class="yiv5912775310gmail_quote">2017-11-28
13:13 GMT-08:00 Paul Edmon <span class="yiv5912775310" dir="ltr"><<a rel="nofollow" shape="rect" class="yiv5912775310" ymailto="mailto:pedmon@cfa.harvard.edu" target="_blank" href="mailto:pedmon@cfa.harvard.edu">pedmon@cfa.harvard.edu</a>></span>:<br clear="none" class="yiv5912775310">
<blockquote class="yiv5912775310gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">So
in an effort to future
proof ourselves we are
trying to build Slurm
against PMIx, but when I
tried to do so I got the
following:<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
Transaction check error:<br clear="none" class="yiv5912775310">
file
/usr/lib64/libpmi.so from
install of
slurm-17.02.9-1fasrc02.el7.cen
tos.x86_64 conflicts with
file from package
pmix-2.0.2-1.el7.centos.x86_64<br clear="none" class="yiv5912775310">
file
/usr/lib64/libpmi2.so from
install of
slurm-17.02.9-1fasrc02.el7.cen
tos.x86_64 conflicts with
file from package
pmix-2.0.2-1.el7.centos.x86_64<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
This is with compiling
Slurm with the
--with-pmix=/usr option.
A few things:<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
1. I'm surprised when I
tell it to use PMIx it
still builds its own
versions of libpmi and
pmi2 given that PMIx
handles that now.<br clear="none" class="yiv5912775310">
</blockquote>
<div class="yiv5912775310"><br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310">PMIx is a plugin and
from multiple perspectives
it makes sense to keep the
other versions available
(i.e. backward compat or
perf comparison) </div>
<div class="yiv5912775310"> </div>
<blockquote class="yiv5912775310gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="yiv5912775310">
2. Does this mean I have
to install PMIx in a
nondefault location? If
so how does that work with
user build codes? I'd
rather not have multiple
versions of PMI around for
people to build against.<br clear="none" class="yiv5912775310">
</blockquote>
<div class="yiv5912775310">When we introduced PMIx
it was in the beta stage
and we didn't want to
build against it by
default. Now it probably
makes sense to assume
--with-pmix by default.</div>
<div class="yiv5912775310">I'm also thinking that
we might need to solve it
at the packagers level by
distributing "slurm-pmix"
package that is builded
and depends on the pmix
package that is currently
shipped with particular
Linux distro.</div>
<div class="yiv5912775310"> </div>
<blockquote class="yiv5912775310gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="yiv5912775310">
3. What is the right way
of building PMIx and Slurm
such that they
interoperate properly?<br clear="none" class="yiv5912775310">
</blockquote>
<div class="yiv5912775310">As for now it is better
to have a PMIx installed
in the well-known
location. And then build
your MPIs or other apps
against this PMIx
installation.</div>
<div class="yiv5912775310">Starting (I think) from
PMIx v2.1 we will have a
cross-version support that
will give some flexibility
about what installation to
use with application,</div>
<div class="yiv5912775310"> </div>
<blockquote class="yiv5912775310gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="yiv5912775310">
Suffice it to say little
to no documentation exists
on how to properly this,
so any guidance would be
much appreciated.</blockquote>
<div class="yiv5912775310">Indeed we have some
problems with the
documentation as PMIx
technology is relatively
new. Hopefully we can fix
this in near future.</div>
<div class="yiv5912775310">Being the original
developer of the PMIx
plugin I'll be happy to
answer any questions and
help to resolve the
issues.</div>
<div class="yiv5912775310yqt4550724288" id="yiv5912775310yqtfd41798">
<div class="yiv5912775310"><br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310"><br clear="none" class="yiv5912775310">
</div>
<div class="yiv5912775310"> </div>
</div>
<blockquote class="yiv5912775310gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><span class="yiv5912775310gmail-HOEnZb"></span>
<div class="yiv5912775310yqt4550724288" id="yiv5912775310yqtfd43154"><br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
-Paul Edmon-</div>
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
</blockquote>
</div>
<br clear="none" class="yiv5912775310">
<br clear="all" class="yiv5912775310">
<div class="yiv5912775310"><br clear="none" class="yiv5912775310">
</div>
-- <br clear="none" class="yiv5912775310">
<div class="yiv5912775310gmail_signature">С
Уважением, Поляков Артем
Юрьевич<br clear="none" class="yiv5912775310">
Best regards, Artem Y.
Polyakov</div>
<div class="yiv5912775310yqt4550724288" id="yiv5912775310yqtfd01302">
</div>
</div>
</div>
</div>
</div>
</div>
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
</div>
</div>
</div>
<div class="yiv5912775310yqt7102444885" id="yiv5912775310yqtfd28013"> </div>
</div>
<div class="yiv5912775310yqt7102444885" id="yiv5912775310yqtfd78525"> </div>
</div>
</div>
</div>
</div>
<br clear="none" class="yiv5912775310">
<br clear="none" class="yiv5912775310">
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br clear="none" class="yiv5912775310">
</div>
<br clear="none" class="yiv5912775310"></div></div></div><br class=""><br class=""></div> </div> </div> </div></div></div></div></blockquote></div><br class=""></div></body></html>