<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Hi,</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">I’m having an issue with srun's --no-alloc flag with mpi which I can reproduce with a fairly simple example. When I run a simple one-core mpi test program as “slurmUser” (the account that
has the --no-alloc privilege) it succeeds:</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">srun -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">However when I add the --no-alloc flag it fails in a way that appears to break mpi (see logfile output and other slurm/mpi info below). It fails similarly on 2 cores. </div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class="">srun --no-alloc -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class="">srun: do not allocate resources</div>
<div class="">srun: error: psana1507: task 0: Exited with exit code 1</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class="">Would anyone have any suggestions for how I could make the “--no-alloc” flag work with mpi? Thanks!</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">chris</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">------------------------------------------------------------------------------------------------------</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Logfile error with --no-alloc flag:</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><br class="">
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class="">
<div class=""><span style="font-family: Courier;" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ more logs/test.log</span></div>
</div>
</div>
<div class="" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">
<div class=""><font face="Courier" class="">--------------------------------------------------------------------------</font></div>
<div class=""><font face="Courier" class="">The application appears to have been direct launched using "srun",</font></div>
<div class=""><font face="Courier" class="">but OMPI was not built with SLURM support. This usually happens</font></div>
<div class=""><font face="Courier" class="">when OMPI was not configured --with-slurm and we weren't able</font></div>
<div class=""><font face="Courier" class="">to discover a SLURM installation in the usual places.</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class="">Please configure as appropriate and try again.</font></div>
<div class=""><font face="Courier" class="">--------------------------------------------------------------------------</font></div>
<div class=""><font face="Courier" class="">*** An error occurred in MPI_Init_thread</font></div>
<div class=""><font face="Courier" class="">*** on a NULL communicator</font></div>
<div class=""><font face="Courier" class="">*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,</font></div>
<div class=""><font face="Courier" class="">*** and potentially your MPI job)</font></div>
<div class=""><font face="Courier" class="">[psana1507:13884] Local abort before MPI_INIT completed completed successfully, </font></div>
<div class=""><font face="Courier" class="">but am not able to aggregate error messages, and not able to guarantee that all </font></div>
<div class=""><font face="Courier" class="">other processes were killed!</font></div>
<div class=""><font face="Courier" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ </font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class="">System information:</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class="">
<div class="">
<div class="">
<div class=""><font face="Courier" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ conda list | grep mpi</font></div>
<div class=""><font face="Courier" class="">mpi 1.0 openmpi conda-forge</font></div>
<div class=""><font face="Courier" class="">mpi4py 3.0.3 py27h9ab638b_1 conda-forge</font></div>
<div class=""><font face="Courier" class="">openmpi 4.1.0 h9b22176_1 conda-forge</font></div>
</div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
</div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class="">
<div class=""><font face="Courier" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --mpi=list</font></div>
<div class=""><font face="Courier" class="">srun: MPI types are...</font></div>
<div class=""><font face="Courier" class="">srun: cray_shasta</font></div>
<div class=""><font face="Courier" class="">srun: none</font></div>
<div class=""><font face="Courier" class="">srun: pmi2</font></div>
<div class=""><font face="Courier" class="">srun: pmix</font></div>
<div class=""><font face="Courier" class="">srun: pmix_v3</font></div>
<div class=""><font face="Courier" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --version</font></div>
<div class=""><font face="Courier" class="">slurm 20.11.3</font></div>
<div class=""><font face="Courier" class="">(ana-4.0.12) psanagpu105:batchtest_slurm$ </font></div>
</div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""><font face="Courier" class=""><br class="">
</font></div>
<div class=""></div>
</font></div>
</div>
</body>
</html>