<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">My apologies - I guess we hadn’t been tracking it that way. I’ll try to add some clarification. We presented a nice table at the BoF and I just need to find a few minutes to post it.<div class=""><br class=""></div><div class="">I believe you do have to build slurm against PMIx so that the pmix plugin is compiled. You then also have to specify --mpi=pmix so slurm knows to use that plugin for this specific job.</div><div class=""><br class=""></div><div class="">You actually might be able to use the PMIx backward compatibility, and you might want to do so with slurm 17.11 and above as Mellanox did a nice job of further optimizing launch performance on IB platforms by adding fabric-based collective implementations to the pmix plugin. If you replace the slurm libpmi and libpmi2 with the ones from PMIx, what will happen is that PMI and PMI2 calls will be converted to their PMIx equivalent and passed to the pmix plugin. This lets you take advantage of what Mellanox did.</div><div class=""><br class=""></div><div class="">The caveat is that your MPI might ask for some PMI/PMI2 feature that we didn’t implement. We have tested with MPICH as well as OMPI and it was fine - but we cannot give you a blanket guarantee (e.g., I’m pretty sure MVAPICH won’t work). Probably safer to stick with the slurm libs for that reason unless you test to ensure it all works.</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 28, 2017, at 6:42 PM, Paul Edmon <<a href="mailto:pedmon@cfa.harvard.edu" class="">pedmon@cfa.harvard.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Okay, I didn't see any note on the PMIx 2.1 page about versions
of slurm it was combatible with so I assumed all of them. My
bad. Thanks for the correction and the help. I just naively used
the rpm spec that was packaged with PMIx which does enable the
legacy support. It seems best then to let PMIx handle pmix solely
and let slurm handle the rest. Thanks!</p><p class="">Am I right in reading that you don't have to build slurm against
PMIx? So it just interoperates with it fine if you just have it
installed and specify pmix as the launch option? That's neat.<br class="">
</p><p class="">-Paul Edmon-<br class="">
</p>
<br class="">
<div class="moz-cite-prefix">On 11/28/2017 6:11 PM, Philip Kovacs
wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:1058339385.5219349.1511910667338@mail.yahoo.com" class="">
<div style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class="">
<div id="yui_3_16_0_ym19_1_1511904995913_53348" class="">Actually if
you're set on installing pmix/pmix-devel from the rpms and
then configuring slurm manually,</div>
<div id="yui_3_16_0_ym19_1_1511904995913_53348" class="">you could just
move the pmix-installed versions of libpmi.so* and libpmi2.so*
to a safe place, configure</div>
<div id="yui_3_16_0_ym19_1_1511904995913_53348" class="">and install
slurm which will drop in its versions pf those libs and then
either use the slurm versions or move</div>
<div id="yui_3_16_0_ym19_1_1511904995913_53348" dir="ltr" class="">the
the pmix versions of libpmi and libpmi2 back into place in
/usr/lib64. </div>
<div class="qtdSeparateBR"><br class="">
<br class="">
</div>
<div class="yahoo_quoted" style="display: block;">
<div style="font-family: Helvetica Neue, Helvetica, Arial,
Lucida Grande, sans-serif; font-size: 13px;" class="">
<div style="font-family: HelveticaNeue, Helvetica Neue,
Helvetica, Arial, Lucida Grande, sans-serif; font-size:
16px;" class="">
<div dir="ltr" class=""><font size="2" face="Arial" class=""> On Tuesday,
November 28, 2017 5:32 PM, Philip Kovacs
<a class="moz-txt-link-rfc2396E" href="mailto:pkdevel@yahoo.com"><pkdevel@yahoo.com></a> wrote:<br class="">
</font></div>
<br class="">
<br class="">
<div class="y_msg_container">
<div id="yiv0414012041" class="">
<div class="">
<div style="background-color: rgb(255, 255, 255); font-family: 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif; font-size: 13px;" class="">
<div id="yiv0414012041" class="">
<div id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_19392" class="">
<div style="background-color: rgb(255, 255, 255); font-size: 13px;" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_19391" class="">
<div id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_14678" class="">This
issue is that pmi 2.0+ provides a
"backward compatibility" feature,
enabled by default, which installs</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_19393" class="">both
libpmi.so and libpmi2.so in addition to
libpmix.so. The route with the least
friction for you would probably</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_19394" class="">be
to uninstall pmix, then install slurm
normally, letting it install its libpmi
and libpmi2. Next configure and compile</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_20421" class="">a
custom pmix with that backward </span>feature
_disabled_, so it only installs
libpmix.so. Slurm will "see" the pmix
library</div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class="">after you install it and load it
via its plugin when you use --mpi=pmix.
Again, just use the Slurm pmi and pmi2
and </div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class="">install pmix separately with the
backward compatible option disabled.</div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span class=""><br class="">
</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_20590" class="">There
is a packaging issue there in which two
packages are trying to install their own
versions of the same files. </span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span class="">That s</span>hould be
brought to attention of the packages.
Meantime you can work around it.</div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span class=""><br class="">
</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span class="">For PMIX:</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><span class=""><br class="">
</span></div>
<div dir="ltr" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_13882" style="" class=""><font id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_20598" face="Courier New, courier, monaco,
monospace, sans-serif" class="">./configure
--disable-pmi-backward-compatibility //
... etc ...</font></div>
<div class="yiv0414012041qtdSeparateBR" id="yiv0414012041yui_3_16_0_ym19_1_1511904995913_19395" style=""><br clear="none" class="">
</div>
</div>
</div>
</div>
<div class="">
<div style="font-family:Helvetica Neue,
Helvetica, Arial, Lucida Grande,
sans-serif;font-size:13px;" class="">
<div style="font-family:HelveticaNeue,
Helvetica Neue, Helvetica, Arial, Lucida
Grande, sans-serif;font-size:16px;" class="">
<div class="yiv0414012041qtdSeparateBR"><br class="">
<br class="">
</div>
<div class="yiv0414012041yqt7102444885" id="yiv0414012041yqtfd45512">
<div dir="ltr" class=""><font size="2" face="Arial" class="">
On Tuesday, November 28, 2017 4:44 PM,
Artem Polyakov
<a class="moz-txt-link-rfc2396E" href="mailto:artpol84@gmail.com"><artpol84@gmail.com></a> wrote:<br clear="none" class="">
</font></div>
<br clear="none" class="">
<br clear="none" class="">
<div class="yiv0414012041y_msg_container">
<div id="yiv0414012041" class="">
<div class="">
<div dir="ltr" class="">Hello, Paul
<div class=""><br clear="none" class="">
</div>
<div class="">Please see below.<br clear="none" class="">
<div class="yiv0414012041gmail_extra"><br clear="none" class="">
<div class="yiv0414012041gmail_quote">2017-11-28
13:13 GMT-08:00 Paul Edmon <span dir="ltr" class=""><<a rel="nofollow" shape="rect" ymailto="mailto:pedmon@cfa.harvard.edu" target="_blank" href="mailto:pedmon@cfa.harvard.edu" moz-do-not-send="true" class="">pedmon@cfa.harvard.edu</a>></span>:<br clear="none" class="">
<blockquote class="yiv0414012041gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex;">So
in an effort to future
proof ourselves we are
trying to build Slurm
against PMIx, but when I
tried to do so I got the
following:<br clear="none" class="">
<br clear="none" class="">
Transaction check error:<br clear="none" class="">
file
/usr/lib64/libpmi.so from
install of
slurm-17.02.9-1fasrc02.el7.cen
tos.x86_64 conflicts with
file from package
pmix-2.0.2-1.el7.centos.x86_64<br clear="none" class="">
file
/usr/lib64/libpmi2.so from
install of
slurm-17.02.9-1fasrc02.el7.cen
tos.x86_64 conflicts with
file from package
pmix-2.0.2-1.el7.centos.x86_64<br clear="none" class="">
<br clear="none" class="">
This is with compiling
Slurm with the
--with-pmix=/usr option.
A few things:<br clear="none" class="">
<br clear="none" class="">
1. I'm surprised when I
tell it to use PMIx it
still builds its own
versions of libpmi and
pmi2 given that PMIx
handles that now.<br clear="none" class="">
</blockquote>
<div class=""><br clear="none" class="">
</div>
<div class="">PMIx is a plugin and
from multiple perspectives
it makes sense to keep the
other versions available
(i.e. backward compat or
perf comparison) </div>
<div class=""> </div>
<blockquote class="yiv0414012041gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="">
2. Does this mean I have
to install PMIx in a
nondefault location? If
so how does that work with
user build codes? I'd
rather not have multiple
versions of PMI around for
people to build against.<br clear="none" class="">
</blockquote>
<div class="">When we introduced PMIx
it was in the beta stage
and we didn't want to
build against it by
default. Now it probably
makes sense to assume
--with-pmix by default.</div>
<div class="">I'm also thinking that
we might need to solve it
at the packagers level by
distributing "slurm-pmix"
package that is builded
and depends on the pmix
package that is currently
shipped with particular
Linux distro.</div>
<div class=""> </div>
<blockquote class="yiv0414012041gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="">
3. What is the right way
of building PMIx and Slurm
such that they
interoperate properly?<br clear="none" class="">
</blockquote>
<div class="">As for now it is better
to have a PMIx installed
in the well-known
location. And then build
your MPIs or other apps
against this PMIx
installation.</div>
<div class="">Starting (I think) from
PMIx v2.1 we will have a
cross-version support that
will give some flexibility
about what installation to
use with application,</div>
<div class=""> </div>
<blockquote class="yiv0414012041gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex;">
<br clear="none" class="">
Suffice it to say little
to no documentation exists
on how to properly this,
so any guidance would be
much appreciated.</blockquote>
<div class="">Indeed we have some
problems with the
documentation as PMIx
technology is relatively
new. Hopefully we can fix
this in near future.</div>
<div class="">Being the original
developer of the PMIx
plugin I'll be happy to
answer any questions and
help to resolve the
issues.</div>
<div class="yiv0414012041yqt4550724288" id="yiv0414012041yqtfd41798">
<div class=""><br clear="none" class="">
</div>
<div class=""><br clear="none" class="">
</div>
<div class=""> </div>
</div>
<blockquote class="yiv0414012041gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex;"><span class="yiv0414012041gmail-HOEnZb"></span>
<div class="yiv0414012041yqt4550724288" id="yiv0414012041yqtfd43154"><br clear="none" class="">
<br clear="none" class="">
-Paul Edmon-</div>
<br clear="none" class="">
<br clear="none" class="">
<br clear="none" class="">
</blockquote>
</div>
<br clear="none" class="">
<br clear="all" class="">
<div class=""><br clear="none" class="">
</div>
-- <br clear="none" class="">
<div class="yiv0414012041gmail_signature">С
Уважением, Поляков Артем
Юрьевич<br clear="none" class="">
Best regards, Artem Y.
Polyakov</div>
<div class="yiv0414012041yqt4550724288" id="yiv0414012041yqtfd01302">
</div>
</div>
</div>
</div>
</div>
</div>
<br clear="none" class="">
<br clear="none" class="">
</div>
</div>
</div>
<div class="yiv0414012041yqt7102444885" id="yiv0414012041yqtfd28013"> </div>
</div>
<div class="yiv0414012041yqt7102444885" id="yiv0414012041yqtfd78525"> </div>
</div>
</div>
</div>
</div>
<br class="">
<br class="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></div></body></html>