<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class="">
<br class=""><div><blockquote type="cite" class=""><div class="">On Dec 7, 2017, at 12:51 PM, Artem Polyakov <<a href="mailto:artpol84@gmail.com" class="">artpol84@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">also please post the output of<div class="">$ srun --mpi=list</div></div></div></blockquote><div><br class=""></div><div><div>[gwolosh@p-slogin bin]$ srun --mpi=list</div><div>srun: MPI types are...</div><div>srun: mpi/mpich1_shmem</div><div>srun: mpi/mpich1_p4</div><div>srun: mpi/lam</div><div>srun: mpi/openmpi</div><div>srun: mpi/none</div><div>srun: mpi/mvapich</div><div>srun: mpi/mpichmx</div><div>srun: mpi/pmi2</div><div>srun: mpi/mpichgm</div></div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">When job crashes - is there any error messages in the relevant slurmd.log's or output on the screen?</div></div></div></blockquote><div><br class=""></div><div>on screen —</div><div><br class=""></div><div><div>[snode4][[274,1],24][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2</div><div>[snode4:5175] *** An error occurred in MPI_Bcast</div><div>[snode4:5175] *** reported by process [17956865,24]</div><div>[snode4:5175] *** on communicator MPI_COMM_WORLD</div><div>[snode4:5175] *** MPI_ERR_OTHER: known error not in list</div><div>[snode4:5175] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,</div><div>[snode4:5175] *** and potentially your MPI job)</div><div>mlx4: local QP operation err (QPN 0005f3, WQE index 40000, vendor syndrome 6c, opcode = 5e)</div><div>srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</div><div>[snode4][[274,1],31][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2</div><div>slurmstepd: error: *** STEP 274.0 ON snode1 CANCELLED AT 2017-12-07T12:55:46 ***</div><div>[snode4:5182] *** An error occurred in MPI_Bcast</div><div>[snode4:5182] *** reported by process [17956865,31]</div><div>[snode4:5182] *** on communicator MPI_COMM_WORLD</div><div>[snode4:5182] *** MPI_ERR_OTHER: known error not in list</div><div>[snode4:5182] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,</div><div>[snode4:5182] *** and potentially your MPI job)</div><div>mlx4: local QP operation err (QPN 0005f7, WQE index 40000, vendor syndrome 6c, opcode = 5e)</div><div>[snode4][[274,1],27][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2</div><div>[snode4:5178] *** An error occurred in MPI_Bcast</div><div>[snode4:5178] *** reported by process [17956865,27]</div><div>[snode4:5178] *** on communicator MPI_COMM_WORLD</div><div>[snode4:5178] *** MPI_ERR_OTHER: known error not in list</div><div>[snode4:5178] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,</div><div>[snode4:5178] *** and potentially your MPI job)</div><div>mlx4: local QP operation err (QPN 0005fa, WQE index 40000, vendor syndrome 6c, opcode = 5e)</div><div>srun: error: snode4: tasks 24,31: Exited with exit code 16</div><div>srun: error: snode4: tasks 25-30: Killed</div><div>srun: error: snode5: tasks 32-39: Killed</div><div>srun: error: snode3: tasks 16-23: Killed</div><div>srun: error: snode8: tasks 56-63: Killed</div><div>srun: error: snode7: tasks 48-55: Killed</div><div>srun: error: snode1: tasks 0-7: Killed</div><div>srun: error: snode2: tasks 8-15: Killed</div><div>srun: error: snode6: tasks 40-47: Killed</div><div><br class=""></div><div>Nothing striking in the slurmd log</div><div><br class=""></div></div><br class=""><blockquote type="cite" class=""><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">2017-12-07 9:49 GMT-08:00 Artem Polyakov <span dir="ltr" class=""><<a href="mailto:artpol84@gmail.com" target="_blank" class="">artpol84@gmail.com</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Hello,<div class=""><br class=""></div><div class="">what is the value of MpiDefault option in your Slurm configuration file?</div></div><div class="gmail_extra"><div class=""><div class="h5"><br class=""><div class="gmail_quote">2017-12-07 9:37 GMT-08:00 Glenn (Gedaliah) Wolosh <span dir="ltr" class=""><<a href="mailto:gwolosh@njit.edu" target="_blank" class="">gwolosh@njit.edu</a>></span>:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hello</div><div class=""><br class=""></div><div class="">This is using Slurm version - 17.02.6 running on Scientific Linux release 7.4 (Nitrogen)</div><div class=""><br class=""></div><div class=""><div class="">[gwolosh@p-slogin bin]$ module li</div><div class=""><br class=""></div><div class="">Currently Loaded Modules:</div><div class=""> 1) GCCcore/.5.4.0 (H) 2) binutils/.2.26 (H) 3) GCC/5.4.0-2.26 4) numactl/2.0.11 5) hwloc/1.11.3 6) OpenMPI/1.10.3</div></div><div class=""><br class=""></div><div class="">If I run</div><div class=""><br class=""></div><div class="">srun --nodes=8 --ntasks-per-node=8 --ntasks=64 ./ep.C.64</div><div class=""><br class=""></div><div class="">It runs successfuly but I get a message —</div><div class=""><br class=""></div><div class=""><div class="">PMI2 initialized but returned bad values for size/rank/jobid.</div><div class="">This is symptomatic of either a failure to use the</div><div class="">"--mpi=pmi2" flag in SLURM, or a borked PMI2 installation.</div><div class="">If running under SLURM, try adding "-mpi=pmi2" to your</div><div class="">srun command line. If that doesn't work, or if you are</div><div class="">not running under SLURM, try removing or renaming the</div><div class="">pmi2.h header file so PMI2 support will not automatically</div><div class="">be built, reconfigure and build OMPI, and then try again</div><div class="">with only PMI1 support enabled.</div><div class=""><br class=""></div><div class="">If I run</div><div class=""><br class=""></div><div class="">srun --nodes=8 --ntasks-per-node=8 --ntasks=64 —mpi=pmi2 ./ep.C.64</div><div class=""><br class=""></div><div class="">The job crashes</div><div class=""><br class=""></div><div class="">If I run via sbatch —</div><div class=""><br class=""></div><div class=""><div class="">#!/bin/bash</div><div class=""># Job name:</div><div class="">#SBATCH --job-name=nas_bench</div><div class="">#SBATCH --nodes=8</div><div class="">#SBATCH --ntasks=64</div><div class="">#SBATCH --ntasks-per-node=8</div><div class="">#SBATCH --time=48:00:00</div><div class="">#SBATCH --output=nas.out.1</div><div class="">#</div><div class="">## Command(s) to run (example):</div><div class="">module use $HOME/easybuild/modules/all/Co<wbr class="">re</div><div class="">module load GCC/5.4.0-2.26 OpenMPI/1.10.3</div><div class="">mpirun -np 64 ./ep.C.64</div></div><div class=""><br class=""></div><div class="">the job crashes</div><div class=""><br class=""></div><div class="">Using easybuild, these are my config options for ompi —</div><div class=""><br class=""></div><div class=""><div class="">configopts = '--with-threads=posix --enable-shared --enable-mpi-thread-multiple --with-verbs '</div><div class="">configopts += '--enable-mpirun-prefix-by-def<wbr class="">ault ' # suppress failure modes in relation to mpirun path</div><div class="">configopts += '--with-hwloc=$EBROOTHWLOC ' # hwloc support</div><div class="">configopts += '--disable-dlopen ' # statically link component, don't do dynamic loading</div><div class="">configopts += '--with-slurm --with-pmi ‘</div></div><div class=""><br class=""></div><div class="">And finally —</div><div class=""><br class=""></div><div class=""><div class="">$ ldd /opt/local/easybuild/software/<wbr class="">Compiler/GCC/5.4.0-2.26/OpenMP<wbr class="">I/1.10.3/bin/orterun | grep pmi</div><div class=""> libpmi.so.0 => /usr/lib64/libpmi.so.0 (0x00007f0129d6d000)</div><div class=""> libpmi2.so.0 => /usr/lib64/libpmi2.so.0 (0x00007f0129b51000)</div></div><div class=""><br class=""></div><div class=""><div class="">$ ompi_info | grep pmi</div><div class=""> MCA db: pmi (MCA v2.0.0, API v1.0.0, Component v1.10.3)</div><div class=""> MCA ess: pmi (MCA v2.0.0, API v3.0.0, Component v1.10.3)</div><div class=""> MCA grpcomm: pmi (MCA v2.0.0, API v2.0.0, Component v1.10.3)</div><div class=""> MCA pubsub: pmi (MCA v2.0.0, API v2.0.0, Component v1.10.3)</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Any suggestions?</div></div><div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class="">_______________<br class="">Gedaliah Wolosh<br class="">IST Academic and Research Computing Systems (ARCS)<br class="">NJIT<br class="">GITC 2203<br class=""><a href="tel:(973)%20596-5437" value="+19735965437" target="_blank" class="">973 596 5437</a><br class=""><a href="mailto:gwolosh@njit.edu" target="_blank" class="">gwolosh@njit.edu</a><br class=""></div></div>
</div>
<br class=""></div></blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div></div></div><span class="HOEnZb"><font color="#888888" class="">-- <br class=""><div class="m_5736140807596716564gmail_signature" data-smartmail="gmail_signature">С Уважением, Поляков Артем Юрьевич<br class="">Best regards, Artem Y. Polyakov</div>
</font></span></div>
</blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature">С Уважением, Поляков Артем Юрьевич<br class="">Best regards, Artem Y. Polyakov</div>
</div>
</div></blockquote></div><br class=""></body></html>