<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.yiv2666596063msonormal, li.yiv2666596063msonormal, div.yiv2666596063msonormal
{mso-style-name:yiv2666596063msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.yiv2666596063msochpdefault, li.yiv2666596063msochpdefault, div.yiv2666596063msochpdefault
{mso-style-name:yiv2666596063msochpdefault;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.yiv2666596063msohyperlink
{mso-style-name:yiv2666596063msohyperlink;}
span.yiv2666596063msohyperlinkfollowed
{mso-style-name:yiv2666596063msohyperlinkfollowed;}
span.yiv2666596063emailstyle17
{mso-style-name:yiv2666596063emailstyle17;}
p.yiv2666596063msonormal1, li.yiv2666596063msonormal1, div.yiv2666596063msonormal1
{mso-style-name:yiv2666596063msonormal1;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.yiv2666596063msonormal2, li.yiv2666596063msonormal2, div.yiv2666596063msonormal2
{mso-style-name:yiv2666596063msonormal2;
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.yiv2666596063msohyperlink1
{mso-style-name:yiv2666596063msohyperlink1;
color:blue;
text-decoration:underline;}
span.yiv2666596063msohyperlinkfollowed1
{mso-style-name:yiv2666596063msohyperlinkfollowed1;
color:purple;
text-decoration:underline;}
span.yiv2666596063emailstyle171
{mso-style-name:yiv2666596063emailstyle171;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.yiv2666596063msochpdefault1, li.yiv2666596063msochpdefault1, div.yiv2666596063msochpdefault1
{mso-style-name:yiv2666596063msochpdefault1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle30
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Philip, Thanks for trying 18.08.8 for me. I finally got a system built with 18.08.8 and I’m having much better success running heterogeneous jobs with PMIX. I haven’t seen the intermittent problem you have - but I’ve just started testing.
I wonder if there is a bug in 19.05.1?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ sinfo -V<o:p></o:p></p>
<p class="MsoNormal">slurm 18.08.8<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">$ srun -wtrek8 -n2 --mpi=pmix : -wtrek9 -n2 mpihh | sort<o:p></o:p></p>
<p class="MsoNormal">srun: job 46073 queued and waiting for resources<o:p></o:p></p>
<p class="MsoNormal">srun: job 46073 has been allocated resources<o:p></o:p></p>
<p class="MsoNormal">Hello world, I am 0 of 4 - running on trek8<o:p></o:p></p>
<p class="MsoNormal">Hello world, I am 1 of 4 - running on trek8<o:p></o:p></p>
<p class="MsoNormal">Hello world, I am 2 of 4 - running on trek9<o:p></o:p></p>
<p class="MsoNormal">Hello world, I am 3 of 4 - running on trek9<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Philip Kovacs<br>
<b>Sent:</b> Tuesday, July 16, 2019 12:03 PM<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] PMIX with heterogeneous jobs<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">Well it looks like it it does fail as often as it works.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">srun --mpi=pmix -n1 -wporthos : -n1 -wathos ./hello<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">srun: job 681 queued and waiting for resources<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">srun: job 681 has been allocated resources<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">slurmstepd: error: athos [0] pmixp_coll_ring.c:613 [pmixp_coll_ring_check] mpi/pmix: ERROR: 0x153ab0017e00: unexpected contrib from athos:0, expected is 1<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">slurmstepd: error: athos [0] pmixp_server.c:930 [_process_server_request] mpi/pmix: ERROR: 0x153ab0017e00: unexpected contrib from athos:0, coll->seq=0, seq=0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">slurmstepd: error: porthos [1] pmixp_coll_ring.c:613 [pmixp_coll_ring_check] mpi/pmix: ERROR: 0x146fdc016bd0: unexpected contrib from porthos:1, expected is 0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif">slurmstepd: error: porthos [1] pmixp_server.c:930 [_process_server_request] mpi/pmix: ERROR: 0x146fdc016bd0: unexpected contrib from porthos:1, coll->seq=0, seq=0<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif"><o:p> </o:p></span></p>
</div>
</div>
<div id="yahoo_quoted_3444529380">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">On Tuesday, July 16, 2019, 09:49:59 AM EDT, Mehlberg, Steve <<a href="mailto:steve.mehlberg@atos.net">steve.mehlberg@atos.net</a>> wrote:
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<div id="yiv2666596063">
<div>
<div>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Has anyone been able to run an MPI job using PMIX and heterogeneous jobs successfully with 19.05 (or even 18.08)? I can run without heterogeneous
jobs but get all sorts of errors when I try and split the job up. <o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">I haven’t used MPI/PMIX much so maybe I’m missing something? Any ideas?
<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"> <o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ sinfo -V</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurm 19.05.1</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"> </span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ which mpicc<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">/opt/openmpi/4.0.1/bin/mpicc<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ sudo yum list pmix</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Installed Packages<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">pmix.x86_64 3.1.2rc1.debug-1.el7 installed
<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"> <o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ mpicc mpihello.c -o mpihh</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"> </span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span lang="ES" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ srun -w trek[8-12] -n5 --mpi=pmix mpihh | sort</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello world, I am 0 of 5 - running on trek8<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello world, I am 1 of 5 - running on trek9<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello world, I am 2 of 5 - running on trek10<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello world, I am 3 of 5 - running on trek11<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Hello world, I am 4 of 5 - running on trek12<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"> <o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$ srun -w trek8 --mpi=pmix : -w trek9 mpihh<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: job 753 queued and waiting for resources<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: job 753 has been allocated resources<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: error: (null) [0] mpi_pmix.c:228 [p_mpi_hook_client_prelaunch] mpi/pmix: ERROR: ot create process mapping<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: error: Application launch failed: MPI plugin's pre-launch setup failed<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: error: Timed out waiting for job step to complete<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_utils.c:457 [pmixp_p2p_send] mpi/pmix: ERROR: send ed, rc=2, exceeded the retry limit<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_server.c:1493 [_slurm_send] mpi/pmix: ERROR: Cannotd message to /var/tmp/sgm-slurm/slurmd.spool/stepd.slurm.pmix.753.0,
size = 649, hostl<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">(null)<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:738 [pmixp_coll_ring_reset_if_to] mpi/p ERROR: 0x7f4f5c016050: collective timeout seq=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll.c:281 [pmixp_coll_log] mpi/pmix: ERROR: Dumpinllective state<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:756 [pmixp_coll_ring_log] mpi/pmix: ERR0x7f4f5c016050: COLL_FENCE_RING state seq=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:758 [pmixp_coll_ring_log] mpi/pmix: ERRmy peerid: 0:trek8<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:765 [pmixp_coll_ring_log] mpi/pmix: ERRneighbor id: next 1:trek9, prev 1:trek9<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:775 [pmixp_coll_ring_log] mpi/pmix: ERRContext ptr=0x7f4f5c0160d0, #0, in-use=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:775 [pmixp_coll_ring_log] mpi/pmix: ERRContext ptr=0x7f4f5c016108, #1, in-use=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:775 [pmixp_coll_ring_log] mpi/pmix: ERRContext ptr=0x7f4f5c016140, #2, in-use=1<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:786 [pmixp_coll_ring_log] mpi/pmix: ERR seq=0 contribs: loc=1/prev=0/fwd=1<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:788 [pmixp_coll_ring_log] mpi/pmix: ERR neighbor contribs [2]:<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:821 [pmixp_coll_ring_log] mpi/pmix: ERR done contrib: -<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:823 [pmixp_coll_ring_log] mpi/pmix: ERR wait contrib: trek9<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:825 [pmixp_coll_ring_log] mpi/pmix: ERR status=PMIXP_COLL_RING_PROGRESS<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_ring.c:829 [pmixp_coll_ring_log] mpi/pmix: ERR buf (offset/size): 553/1659<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1317 [pmixp_coll_tree_reset_if_to] mpi/: ERROR: 0x7f4f5c01eb90: collective timeout seq=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll.c:281 [pmixp_coll_log] mpi/pmix: ERROR: Dumpinllective state<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1336 [pmixp_coll_tree_log] mpi/pmix: ER 0x7f4f5c01eb90: COLL_FENCE_TREE state seq=0 contribs: loc=1/prnt=0/child=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1338 [pmixp_coll_tree_log] mpi/pmix: ER my peerid: 0:trek8<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1341 [pmixp_coll_tree_log] mpi/pmix: ER root host: 0:trek8<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1355 [pmixp_coll_tree_log] mpi/pmix: ER child contribs [1]:<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1382 [pmixp_coll_tree_log] mpi/pmix: ER done contrib: -<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1384 [pmixp_coll_tree_log] mpi/pmix: ER wait contrib: trek9<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1391 [pmixp_coll_tree_log] mpi/pmix: ER status: coll=COLL_COLLECT upfw=COLL_SND_NONE dfwd=COLL_SND_NONE<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1393 [pmixp_coll_tree_log] mpi/pmix: ER dfwd status: dfwd_cb_cnt=0, dfwd_cb_wait=0<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_coll_tree.c:1396 [pmixp_coll_tree_log] mpi/pmix: ER bufs (offset/size): upfw 91/16415, dfwd 64/16415<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: trek8 [0] pmixp_dmdx.c:466 [pmixp_dmdx_timeout_cleanup] mpi/pmix: E: timeout: ns=slurm.pmix.753.0, rank=1, host=trek9, ts=1563206701<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] pml_ucx.c:176 Error: Failed to receive UCX worker address: Not found (-<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] pml_ucx.c:447 Error: Failed to resolve UCX endpoint for rank 1<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** An error occurred in MPI_Barrier<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** reported by process [1543534944,0]<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** on communicator MPI_COMM_WORLD<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** MPI_ERR_OTHER: known error not in list<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[trek8:23061] *** and potentially your MPI job)<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">slurmstepd: error: *** STEP 753.0 ON trek8 CANCELLED AT 2019-07-15T09:05:01 ***<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: Job step aborted: Waiting up to 32 seconds for job step to finish.<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">srun: error: trek8: task 0: Exited with exit code 16<o:p></o:p></span></p>
<p class="yiv2666596063msonormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">[slurm@trek8 mpihello]$<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>