<div dir="ltr"><div dir="ltr">Hmmm, I can't quite replicate that:<div><br></div><div><br></div><div><div>dmj@cori11:~> salloc -C knl -q interactive -N 2 --no-shell</div><div>salloc: Granted job allocation 18219715</div><div>salloc: Waiting for resource configuration</div><div>salloc: Nodes nid0[2318-2319] are ready for job</div><div>dmj@cori11:~> srun --jobid=18219715 /bin/false</div><div>srun: error: nid02318: task 0: Exited with exit code 1</div><div>srun: Terminating job step 18219715.0</div><div>srun: error: nid02319: task 1: Exited with exit code 1</div><div>dmj@cori11:~> echo $?</div><div>1</div><div>dmj@cori11:~> squeue -u dmj</div><div> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</div><div> 18219715 interacti (null) dmj R 0:57 2 nid0[2318-2319]</div><div>dmj@cori11:~> srun --jobid=18219715 /bin/false</div><div>srun: error: nid02319: task 1: Exited with exit code 1</div><div>srun: Terminating job step 18219715.1</div><div>srun: error: nid02318: task 0: Exited with exit code 1</div><div>dmj@cori11:~> squeue -u dmj</div><div> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</div><div> 18219715 interacti (null) dmj R 1:17 2 nid0[2318-2319]</div><div>dmj@cori11:~></div><div><br></div><div><br></div><div><br></div><div>Is it possible that your failing sruns are not properly terminating when the first rank crashes and is actually consuming all the requested time?</div><div><br></div><div>-Doug</div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font face="courier new, monospace" size="1">----</font></div><div><font face="arial, helvetica, sans-serif"><font size="2">Doug Jacobsen, Ph.D.</font><br></font></div><div><font face="arial, helvetica, sans-serif" size="1">NERSC Computer Systems Engineer</font></div><div><font face="arial, helvetica, sans-serif" size="1">Acting Group Lead, Computational Systems Group</font></div><div><font size="1"><font face="arial, helvetica, sans-serif"><a href="http://www.nersc.gov" target="_blank">National Energy Research Scientific Computing Center</a></font><br></font></div><div><font size="1"><a href="mailto:dmjacobsen@lbl.gov" style="font-family:arial,helvetica,sans-serif" target="_blank">dmjacobsen@lbl.gov</a><span style="font-family:arial,helvetica,sans-serif"></span></font><br></div><div><font size="1"><br></font></div><div><font size="1"><font face="courier new, monospace"><span style="color:rgb(136,136,136)">------------- __o</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">---------- _ '\<,_</span><br style="color:rgb(136,136,136)"><span style="color:rgb(136,136,136)">----------(_)/ (_)__________________________</span></font><br></font></div><div style="font-family:arial;font-size:small"><br></div></div></div></div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 24, 2019 at 9:24 AM Pritchard Jr., Howard <<a href="mailto:howardp@lanl.gov">howardp@lanl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Hello Slurm experts,</div>
<div><br>
</div>
<div>We have a workflow where we have a script which invoke salloc —noshell and then launches a series of MPI</div>
<div>jobs using srun with the jobid= option to make use of the reservation we got from the salloc invocation.</div>
<div>We are needing to do things this way because the script itself needs to report back the results of the</div>
<div>tests to an external server running at AWS. The compute nodes within the allocated partition have no connectivity</div>
<div>to the internet, hence our use of the —noshell option.</div>
<div><br>
</div>
<div>This is all fine except for an annoying behavior of slurm. If we have no test failures, I.e. all srun’ed tests</div>
<div>exist successfully everything works fine. However, once we start having failed tests, and hence non zero</div>
<div>status return from srun, we maybe get one or two tests to run, and then slurm cancels the reservation.</div>
<div><br>
</div>
<div>Here’s an example output from the script as its running some MPI tests, then some fail, then slurm drops</div>
<div>our reservation:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd start: srun -n 16 -c 4 --mpi=pmix --jobid=2974 /users/foobar/runInAllocMTT/mtt/masterWa</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">lloc_scratch/TestGet_IBM/ompi-tests/ibm/random/leakcatch</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: seed value: -219475876</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 0</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 1</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 2</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 3</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 4</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 5</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 6</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 7</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 8</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 9</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 10</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 11</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 12</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 13</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 14</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 15</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 16</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 17</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 18</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 19</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: 20</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stdout: ERROR: buf 778 element 749856 is 103 should be 42</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: --------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: with errorcode 16.</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: </span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: You may or may not see output from other processes, depending on</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: exactly when Open MPI kills them.</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: --------------------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: slurmstepd: error: *** STEP 2974.490 ON st03 CANCELLED AT 2019-01-22T20:02:22 ***</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: error: st03: task 0: Exited with exit code 16</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: error: st03: tasks 1-15: Killed</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd done</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd start: srun -n 16 -c 4 --mpi=pmix --jobid=2974 /users/foobar/runInAllocMTT/mtt/masterWa</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">lloc_scratch/TestGet_IBM/ompi-tests/ibm/random/maxsoak</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: Job step aborted: Waiting up to 32 seconds for job step to finish.</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: slurmstepd: error: *** STEP 2974.491 ON st03 CANCELLED AT 2019-01-22T23:06:08 DUE TO TIME LI</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">MIT ***</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: error: st03: tasks 0-15: Terminated</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd done</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd start: srun -n 16 -c 4 --mpi=pmix --jobid=2974 /users/foobar/runInAllocMTT/mtt/masterWa</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">lloc_scratch/TestGet_IBM/ompi-tests/ibm/random/op_commutative</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">stderr: srun: error: Unable to allocate resources: Invalid job id specified</span></p>
<p style="margin:0px;font-size:15px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">
<span style="font-variant-ligatures:no-common-ligatures">ExecuteCmd done</span></p>
</div>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>This is not due to the allocation being revoked due to a time limit, even though the message says such. The job had been running only about 30 minutes</div>
<div>into a 3 hour reservation. We’ve double checked that and on one cluster which we can configure, we set the default</div>
<div>job timelimit to infinite and still observe the issue. But the fact that SLURM is reporting its a TIMELIMIT thing may be hinting at what’s going on that</div>
<div>SLURM revokes the allocation.</div>
<div><br>
</div>
<div>We see this on every cluster we’ve tried so far, so it doesn’t appear to be a site-specific configuration issue. </div>
<div><br>
</div>
<div>Any insights into how to workaround/fix this problem would be appreciated.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Howard</div>
<div><br>
</div>
<div><br>
</div>
<div>-- </div>
<div>
<div>
<div>Howard Pritchard</div>
</div>
<div>B Schedule</div>
<div>HPC-ENV</div>
<div>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt">Office 9, 2<sup>nd</sup> floor Research Park<u></u><u></u></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt">TA-03, Building 4200, Room 203</p>
</div>
<div>Los Alamos National Laboratory</div>
<div><br>
</div>
</div>
</div>
</blockquote></div>