[slurm-users] --no-alloc breaks mpi?

[O'Grady, Paul Christopher]

Hi,

I’m having an issue with srun's --no-alloc flag with mpi which I can reproduce with a fairly simple example.  When I run a simple one-core mpi test program as “slurmUser” (the account that has the --no-alloc privilege) it succeeds:

srun -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py

However when I add the --no-alloc flag it fails in a way that appears to break mpi (see logfile output and other slurm/mpi info below).  It fails similarly on 2 cores.

srun --no-alloc -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py
srun: do not allocate resources
srun: error: psana1507: task 0: Exited with exit code 1

Would anyone have any suggestions for how I could make the “--no-alloc” flag work with mpi?  Thanks!

chris

------------------------------------------------------------------------------------------------------

Logfile error with --no-alloc flag:

(ana-4.0.12) psanagpu105:batchtest_slurm$ more logs/test.log
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM support. This usually happens
when OMPI was not configured --with-slurm and we weren't able
to discover a SLURM installation in the usual places.

Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[psana1507:13884] Local abort before MPI_INIT completed completed successfully,
but am not able to aggregate error messages, and not able to guarantee that all
other processes were killed!
(ana-4.0.12) psanagpu105:batchtest_slurm$

System information:

(ana-4.0.12) psanagpu105:batchtest_slurm$ conda list | grep mpi
mpi                       1.0                     openmpi    conda-forge
mpi4py                    3.0.3            py27h9ab638b_1    conda-forge
openmpi                   4.1.0                h9b22176_1    conda-forge


(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --mpi=list
srun: MPI types are...
srun: cray_shasta
srun: none
srun: pmi2
srun: pmix
srun: pmix_v3
(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --version
slurm 20.11.3
(ana-4.0.12) psanagpu105:batchtest_slurm$



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210308/1f8c6aa3/attachment-0001.htm>