<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>I am facing the same problem that was quoted long ago (2019) in this mailing mailing reference:</p>
<p><br>
</p>
<p><a href="https://lists.schedmd.com/pipermail/slurm-users/2019-July/003785.html" class="OWAAutoLink">https://lists.schedmd.com/pipermail/slurm-users/2019-July/003785.html</a><br>
</p>
<p><br>
</p>
<p>but with more recent version of slurm i.e:</p>
<p><br>
</p>
<p></p>
<div>slurm 21.08.8-2</div>
<div>PMIx 2.2.5 (p<span style="font-size: 12pt;">mix-2.2.5-1.el8.src.rpm)</span></div>
<div>openMPI 4.1.5</div>
<div><br>
</div>
<div>In a similar way to my predecessor, running MPI heterogeneous jobs (OSU benchmarks) using this</div>
<div>slurm+PMIx version installed on the host gives sporadically this type of error</div>
<div><br>
</div>
<div>>>></div>
<div><span>slurmstepd: error: mpi/pmix_v2: _tcp_connect: lxbk1177 [0]: pmixp_dconn_tcp.c:139: Cannot establish the connection</span><br>
</div>
<div><span>slurmstepd: error: mpi/pmix_v2: pmixp_dconn_connect: lxbk1177 [0]: pmixp_dconn.h:246: Cannot establish direct connection to lxbk1177 (0)</span></div>
<div>
<div>slurmstepd: error: mpi/pmix_v2: _process_extended_hdr: lxbk1177 [0]: pmixp_server.c:738: Unable to connect to 0</div>
<div>
<div>slurmstepd: error: mpi/pmix_v2: pmixp_coll_ring_check: lxbk1177 [0]: pmixp_coll_ring.c:618: 0x14cd84047ab0: unexpected contrib from lxbk1177:0, expected is 1</div>
<div>
<div>slurmstepd: error: mpi/pmix_v2: _process_server_request: lxbk1177 [0]: pmixp_server.c:942: 0x14cd84047ab0: unexpected contrib from lxbk1177:0, coll->seq=0, seq=0</div>
>>></div>
</div>
<div><br>
</div>
<div>So very similar problem indeed.</div>
<div>Additionally when the jobs completes, from time to time it cannot finish properly and stay in RUNNING state an one needs to manually</div>
<div>cancel the job.</div>
<div><br>
</div>
<div>Is the hetjob functionality really supporting this case?</div>
<div>If yes, any ideas what can be wrong here?</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>Job submission details:</div>
<div>==================</div>
<div><br>
</div>
<p></p>
<p>- submit script:</p>
<p><span>sbatch --ntasks 1 --ntasks-per-core 1 --cpus-per-task 2 -p main -D ./data -o %j.out.log -e %j.err.log : --ntasks 1 --ntasks-per-core 1 --cpus-per-task 1 -p main -D ./data -o %j.out.log -e %j.err.log ./run-file.sh</span><br>
</p>
<p><span><br>
</span></p>
<p><span><br>
</span></p>
<p><span>- run-file.sh:</span></p>
<p><span> </span></p>
<p><span>export CONT=<std_singularity_container>.sif</span></p>
<p><span>srun -vv --mpi=pmix --export=ALL : $CONT collective/osu_allreduce -f -i 100 -x 10</span><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size: 12pt; color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif, EmojiFont, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols;">
<p></p>
<div><br>
</div>
<div><span style="font-size:9pt">---------</span><span style="font-size:9pt"></span></div>
<div><span style="font-size:9pt">Denis Bertini</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Abteilung: CIT</span><br>
<span style="font-size:9pt"></span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Ort: SB3 2.265a</span></div>
<span style="font-size:9pt"></span>
<div><br>
<span style="font-size:9pt"></span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Tel: +49 6159 71 2240</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Fax: +49 6159 71 2986</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">E-Mail: d.bertini@gsi.de</span></div>
<span style="font-size:9pt"></span>
<div><br>
<span style="font-size:9pt"></span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">GSI Helmholtzzentrum für Schwerionenforschung GmbH</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Planckstraße 1, 64291 Darmstadt, Germany, www.gsi.de</span></div>
<span style="font-size:9pt"></span>
<div><br>
<span style="font-size:9pt"></span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Commercial Register / Handelsregister: Amtsgericht Darmstadt, HRB 1528</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Managing Directors / Geschäftsführung:</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Professor Dr. Paolo Giubellino, Dr. Ulrich Breuer, Jörg Blaurock</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Chairman of the GSI Supervisory Board / Vorsitzender des GSI-Aufsichtsrats:</span></div>
<span style="font-size:9pt"></span>
<div><span style="font-size:9pt">Ministerialdirigent Dr. Volkmar Dietz</span></div>
<p></p>
</div>
</div>
</div>
</body>
</html>