<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Courier;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi Chris,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What’s happening is that there’s no SLURM_JOBID (my speculation since I don’t have perms to use –no-alloc) is set, but SLURM_NODELIST may be set, so its confusing ORTE.<o:p></o:p></p>
<p class="MsoNormal">Could you list which SLURM env variables are set in the shell in which your running the srun command?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Howard<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "O'Grady, Paul Christopher" <cpo@slac.stanford.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Monday, March 8, 2021 at 2:09 PM<br>
<b>To: </b>"slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>[EXTERNAL] [slurm-users] --no-alloc breaks mpi?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">I’m having an issue with srun's --no-alloc flag with mpi which I can reproduce with a fairly simple example. When I run a simple one-core mpi test program as “slurmUser” (the account that
has the --no-alloc privilege) it succeeds:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">srun -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">However when I add the --no-alloc flag it fails in a way that appears to break mpi (see logfile output and other slurm/mpi info below). It fails similarly on 2 cores. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">srun --no-alloc -p psfehq -n 1 -o logs/test.log -w psana1507 python ~/ipsana/mpi_simpletest.py<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">srun: do not allocate resources<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">srun: error: psana1507: task 0: Exited with exit code 1<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Would anyone have any suggestions for how I could make the “--no-alloc” flag work with mpi? Thanks!<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">chris<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">------------------------------------------------------------------------------------------------------<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black">Logfile error with --no-alloc flag:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ more logs/test.log</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">--------------------------------------------------------------------------</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">The application appears to have been direct launched using "srun",</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">but OMPI was not built with SLURM support. This usually happens</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">when OMPI was not configured --with-slurm and we weren't able</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">to discover a SLURM installation in the usual places.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">Please configure as appropriate and try again.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">--------------------------------------------------------------------------</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">*** An error occurred in MPI_Init_thread</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">*** on a NULL communicator</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">*** and potentially your MPI job)</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">[psana1507:13884] Local abort before MPI_INIT completed completed successfully, </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">but am not able to aggregate error messages, and not able to guarantee that all </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">other processes were killed!</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">System information:</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ conda list | grep mpi<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">mpi 1.0 openmpi conda-forge<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">mpi4py 3.0.3 py27h9ab638b_1 conda-forge<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">openmpi 4.1.0 h9b22176_1 conda-forge<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --mpi=list<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: MPI types are...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: cray_shasta<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: none<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: pmi2<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: pmix<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">srun: pmix_v3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ srun --version<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">slurm 20.11.3<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black">(ana-4.0.12) psanagpu105:batchtest_slurm$ <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:Courier;color:black"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>