[slurm-users] NAS benchmarks - problem with openmpi, slurm and pmi

Glenn (Gedaliah) Wolosh gwolosh at njit.edu
Thu Dec 7 13:53:56 MST 2017



> On Dec 7, 2017, at 3:26 PM, Artem Polyakov <artpol84 at gmail.com> wrote:
> 
> Given that ring is working I don't think that it's a PMI problem.
> 
> Can you try running NPB with the tcp btl parameters that I've provided? (I assume you have TCP interconnect, let me know if it's not a case).
> 
> чт, 7 дек. 2017 г. в 12:03, Glenn (Gedaliah) Wolosh <gwolosh at njit.edu <mailto:gwolosh at njit.edu>>:
>> On Dec 7, 2017, at 1:18 PM, Artem Polyakov <artpol84 at gmail.com <mailto:artpol84 at gmail.com>> wrote:
>> 
>> Couple of things to try to locate the issue:
>> 
>> 1. To make sure that PMI is not working: have you tried to run something simple (like hello_world (https://github.com/open-mpi/ompi/blob/master/examples/hello_c.c <https://github.com/open-mpi/ompi/blob/master/examples/hello_c.c>) and ring (https://github.com/open-mpi/ompi/blob/master/examples/ring_c.c <https://github.com/open-mpi/ompi/blob/master/examples/ring_c.c>). Please try to run those two and post the results.
>> 2. If hello is working and ring is not can you try to change the fabric to TCP: 
>> $ export OMPI_MCA_btl=tcp,self
>> $ export OMPI_MCA_pml=ob1
>> $ srun ...
>> 
>> Please provide the outputs


export OMPI_MCA_btl=tcp,self
export OMPI_MCA_pml=ob1

srun --nodes=8 --ntasks-per-node=8 --ntasks=64 --mpi=pmi2 ./ep.C.64

This works —

AS Parallel Benchmarks 3.3 -- EP Benchmark

 Number of random numbers generated:      8589934592
 Number of active processes:                      64

EP Benchmark Results:

CPU Time =    5.9208
N = 2^   32
No. Gaussian Pairs =    3373275903.
Sums =     4.764367927992081D+04   -8.084072988045549D+04
Counts:
  0    1572172634.
  1    1501108549.
  2     281805648.
  3      17761221.
  4        424017.
  5          3821.
  6            13.
  7             0.
  8             0.
  9             0.


 EP Benchmark Completed.
 Class           =                        C
 Size            =               8589934592
 Iterations      =                        0
 Time in seconds =                     5.92
 Total processes =                       64
 Compiled procs  =                       64
 Mop/s total     =                  1450.82
 Mop/s/process   =                    22.67
 Operation type  = Random numbers generated
 Verification    =               SUCCESSFUL
 Version         =                    3.3.1
 Compile date    =              07 Dec 2017

 Compile options:
    MPIF77       = mpif77
    FLINK        = $(MPIF77)
    FMPI_LIB     = -L/opt/local/easybuild/software/Compiler/GC...
    FMPI_INC     = -I/opt/local/easybuild/software/Compiler/GC...
    FFLAGS       = -O
    FLINKFLAGS   = -O
    RAND         = randi8


 Please send feedbacks and/or the results of this run to:

 NPB Development Team
 Internet: npb at nas.nasa.gov

Hmm...

> srun --mpi=pmi2 --ntasks-per-node=8 --ntasks=16 ./hello_c > hello_c.out
> 
> Hello, world, I am 24 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 0 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 25 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 1 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 27 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 2 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 29 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 31 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 30 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 4 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 5 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 17 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 3 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 7 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 6 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 18 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 22 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 23 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 19 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 9 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 20 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 8 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 10 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 13 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 11 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 26 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 16 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 14 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 28 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 21 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 15 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> Hello, world, I am 12 of 32, (Open MPI v1.10.3, package: Open MPI gwolosh at snode2.p-stheno.tartan.njit.edu <mailto:gwolosh at snode2.p-stheno.tartan.njit.edu> Distribution, ident: 1.10.3, repo rev: v1.10.2-251-g9acf492, Jun 14, 2016, 150)
> 
>  srun --mpi=pmi2 --ntasks-per-node=8 --ntasks=16 --nodes=2 ./ring_c > ring_c.out
> 
> Process 1 exiting
> Process 12 exiting
> Process 14 exiting
> Process 13 exiting
> Process 3 exiting
> Process 11 exiting
> Process 5 exiting
> Process 6 exiting
> Process 2 exiting
> Process 4 exiting
> Process 9 exiting
> Process 10 exiting
> Process 7 exiting
> Process 15 exiting
> Process 0 sending 10 to 1, tag 201 (16 processes in ring)
> Process 0 sent to 1
> Process 0 decremented value: 9
> Process 0 decremented value: 8
> Process 0 decremented value: 7
> Process 0 decremented value: 6
> Process 0 decremented value: 5
> Process 0 decremented value: 4
> Process 0 decremented value: 3
> Process 0 decremented value: 2
> Process 0 decremented value: 1
> Process 0 decremented value: 0
> Process 0 exiting
> Process 8 exiting
> 
>> 
>> 2017-12-07 10:05 GMT-08:00 Glenn (Gedaliah) Wolosh <gwolosh at njit.edu <mailto:gwolosh at njit.edu>>:
>> 
>> 
>>> On Dec 7, 2017, at 12:51 PM, Artem Polyakov <artpol84 at gmail.com <mailto:artpol84 at gmail.com>> wrote:
>>> 
>>> also please post the output of
>>> $ srun --mpi=list
>> 
>> [gwolosh at p-slogin bin]$ srun --mpi=list
>> srun: MPI types are...
>> srun: mpi/mpich1_shmem
>> srun: mpi/mpich1_p4
>> srun: mpi/lam
>> srun: mpi/openmpi
>> srun: mpi/none
>> srun: mpi/mvapich
>> srun: mpi/mpichmx
>> srun: mpi/pmi2
>> srun: mpi/mpichgm
>> 
>> 
>>> 
>>> When job crashes - is there any error messages in the relevant slurmd.log's or output on the screen?
>> 
>> on screen —
>> 
>> [snode4][[274,1],24][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2
>> [snode4:5175] *** An error occurred in MPI_Bcast
>> [snode4:5175] *** reported by process [17956865,24]
>> [snode4:5175] *** on communicator MPI_COMM_WORLD
>> [snode4:5175] *** MPI_ERR_OTHER: known error not in list
>> [snode4:5175] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> [snode4:5175] ***    and potentially your MPI job)
>> mlx4: local QP operation err (QPN 0005f3, WQE index 40000, vendor syndrome 6c, opcode = 5e)
>> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
>> [snode4][[274,1],31][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2
>> slurmstepd: error: *** STEP 274.0 ON snode1 CANCELLED AT 2017-12-07T12:55:46 ***
>> [snode4:5182] *** An error occurred in MPI_Bcast
>> [snode4:5182] *** reported by process [17956865,31]
>> [snode4:5182] *** on communicator MPI_COMM_WORLD
>> [snode4:5182] *** MPI_ERR_OTHER: known error not in list
>> [snode4:5182] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> [snode4:5182] ***    and potentially your MPI job)
>> mlx4: local QP operation err (QPN 0005f7, WQE index 40000, vendor syndrome 6c, opcode = 5e)
>> [snode4][[274,1],27][connect/btl_openib_connect_udcm.c:1448:udcm_wait_for_send_completion] send failed with verbs status 2
>> [snode4:5178] *** An error occurred in MPI_Bcast
>> [snode4:5178] *** reported by process [17956865,27]
>> [snode4:5178] *** on communicator MPI_COMM_WORLD
>> [snode4:5178] *** MPI_ERR_OTHER: known error not in list
>> [snode4:5178] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
>> [snode4:5178] ***    and potentially your MPI job)
>> mlx4: local QP operation err (QPN 0005fa, WQE index 40000, vendor syndrome 6c, opcode = 5e)
>> srun: error: snode4: tasks 24,31: Exited with exit code 16
>> srun: error: snode4: tasks 25-30: Killed
>> srun: error: snode5: tasks 32-39: Killed
>> srun: error: snode3: tasks 16-23: Killed
>> srun: error: snode8: tasks 56-63: Killed
>> srun: error: snode7: tasks 48-55: Killed
>> srun: error: snode1: tasks 0-7: Killed
>> srun: error: snode2: tasks 8-15: Killed
>> srun: error: snode6: tasks 40-47: Killed
>> 
>> Nothing striking in the slurmd log
>> 
>> 
>>> 
>>> 2017-12-07 9:49 GMT-08:00 Artem Polyakov <artpol84 at gmail.com <mailto:artpol84 at gmail.com>>:
>>> Hello,
>>> 
>>> what is the value of MpiDefault option in your Slurm configuration file?
>>> 
>>> 2017-12-07 9:37 GMT-08:00 Glenn (Gedaliah) Wolosh <gwolosh at njit.edu <mailto:gwolosh at njit.edu>>:
>>> Hello
>>> 
>>> This is using Slurm version - 17.02.6 running on Scientific Linux release 7.4 (Nitrogen)
>>> 
>>> [gwolosh at p-slogin bin]$ module li
>>> 
>>> Currently Loaded Modules:
>>>   1) GCCcore/.5.4.0 (H)   2) binutils/.2.26 (H)   3) GCC/5.4.0-2.26   4) numactl/2.0.11   5) hwloc/1.11.3   6) OpenMPI/1.10.3
>>> 
>>> If I run
>>> 
>>> srun --nodes=8 --ntasks-per-node=8 --ntasks=64  ./ep.C.64
>>> 
>>> It runs successfuly but I get a message —
>>> 
>>> PMI2 initialized but returned bad values for size/rank/jobid.
>>> This is symptomatic of either a failure to use the
>>> "--mpi=pmi2" flag in SLURM, or a borked PMI2 installation.
>>> If running under SLURM, try adding "-mpi=pmi2" to your
>>> srun command line. If that doesn't work, or if you are
>>> not running under SLURM, try removing or renaming the
>>> pmi2.h header file so PMI2 support will not automatically
>>> be built, reconfigure and build OMPI, and then try again
>>> with only PMI1 support enabled.
>>> 
>>> If I run
>>> 
>>> srun --nodes=8 --ntasks-per-node=8 --ntasks=64  —mpi=pmi2 ./ep.C.64
>>> 
>>> The job crashes
>>> 
>>> If I run via sbatch —
>>> 
>>> #!/bin/bash
>>> # Job name:
>>> #SBATCH --job-name=nas_bench
>>> #SBATCH --nodes=8
>>> #SBATCH --ntasks=64
>>> #SBATCH --ntasks-per-node=8
>>> #SBATCH --time=48:00:00
>>> #SBATCH --output=nas.out.1
>>> #
>>> ## Command(s) to run (example):
>>> module use $HOME/easybuild/modules/all/Core
>>> module load GCC/5.4.0-2.26 OpenMPI/1.10.3
>>> mpirun -np 64  ./ep.C.64
>>> 
>>> the job crashes
>>> 
>>> Using easybuild, these are my config options for ompi —
>>> 
>>> configopts = '--with-threads=posix --enable-shared --enable-mpi-thread-multiple --with-verbs '
>>> configopts += '--enable-mpirun-prefix-by-default '  # suppress failure modes in relation to mpirun path
>>> configopts += '--with-hwloc=$EBROOTHWLOC '  # hwloc support
>>> configopts += '--disable-dlopen '  # statically link component, don't do dynamic loading
>>> configopts += '--with-slurm --with-pmi ‘
>>> 
>>> And finally —
>>> 
>>> $ ldd /opt/local/easybuild/software/Compiler/GCC/5.4.0-2.26/OpenMPI/1.10.3/bin/orterun | grep pmi
>>>         libpmi.so.0 => /usr/lib64/libpmi.so.0 (0x00007f0129d6d000)
>>>         libpmi2.so.0 => /usr/lib64/libpmi2.so.0 (0x00007f0129b51000)
>>> 
>>> $ ompi_info | grep pmi
>>>                   MCA db: pmi (MCA v2.0.0, API v1.0.0, Component v1.10.3)
>>>                  MCA ess: pmi (MCA v2.0.0, API v3.0.0, Component v1.10.3)
>>>              MCA grpcomm: pmi (MCA v2.0.0, API v2.0.0, Component v1.10.3)
>>>               MCA pubsub: pmi (MCA v2.0.0, API v2.0.0, Component v1.10.3)
>>> 
>>> 
>>> Any suggestions?
>>> _______________
>>> Gedaliah Wolosh
>>> IST Academic and Research Computing Systems (ARCS)
>>> NJIT
>>> GITC 2203
>>> 973 596 5437 <tel:(973)%20596-5437>
>>> gwolosh at njit.edu <mailto:gwolosh at njit.edu>
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> С Уважением, Поляков Артем Юрьевич
>>> Best regards, Artem Y. Polyakov
>>> 
>>> 
>>> 
>>> -- 
>>> С Уважением, Поляков Артем Юрьевич
>>> Best regards, Artem Y. Polyakov
>> 
>> 
>> 
>> 
>> -- 
>> С Уважением, Поляков Артем Юрьевич
>> Best regards, Artem Y. Polyakov
> 
> -- 
> ----- Best regards, Artem Polyakov (Mobile mail)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20171207/bb88f70b/attachment-0001.html>


More information about the slurm-users mailing list