<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body smarttemplateinserted="true">
It appears that we have gotten to the bottom of this problem! We
discovered that we only seem to see this problem if our overnight
test script is run with "nohup," as we have been doing for several
years. Typically, we would see the mysterious cancellations about
once every other day, or 3-4 times a week. In the week+ since we
started using "tmux" instead, we haven't seen this problem at all.<br>
<br>
On that basis, I'm declaring success!<br>
<br>
Many thanks to Doug Meyer and Chris Samuel for jumping in to offer
suggestions.<br>
<br>
Andy<br>
<br>
<div id="smartTemplate4-quoteHeader">
<hr> <b>From:</b> Andy Riebs <a class="moz-txt-link-rfc2396E" href="mailto:andy.riebs@hpe.com"><andy.riebs@hpe.com></a> <br>
<b>Sent:</b> Thursday, January 31, 2019 2:04PM <br>
<b>To:</b> Slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@schedmd.com"><slurm-users@schedmd.com></a><br>
<b>Cc:</b> <br>
<b>Subject:</b> Mysterious job terminations on Slurm 17.11.10 <br>
</div>
<div class="replaced-blockquote"
cite="mid:e1662bf4-f608-2fc4-6591-2b2602e17de8@hpe.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hi All,<br>
<br>
Just checking to see if this sounds familiar to anyone.<br>
<br>
Environment:<br>
- CentOS 7.5 x86_64<br>
- Slurm 17.11.10 (but this also happened with 17.11.5)<br>
<br>
We typically run about 100 tests/night, selected from a handful of
favorites. For roughly 1 in 300 test runs, we see one of two
mysterious failures:<br>
<br>
1. The 5 minute cancellation<br>
<br>
A job will be rolling along, generating it's expected output, and
then this message appears:<br>
<blockquote>srun: forcing job termination<br>
srun: Job step aborted: Waiting up to 32 seconds for job step to
finish.<br>
slurmstepd: error: *** STEP 3531.0 ON nodename CANCELLED AT
2019-01-30T07:35:50 ***<br>
srun: error: nodename: task 250: Terminated<br>
srun: Terminating job step 3531.0<br>
</blockquote>
sacct reports<br>
<blockquote><tt> JobID Start
End ExitCode State </tt><br>
<tt>------------ ------------------- -------------------
-------- ---------- </tt><br>
<tt>3418 2019-01-29T05:54:07 2019-01-29T05:59:16
0:9 FAILED</tt><br>
</blockquote>
These failures consistently happen at just about 5 minutes into
the run when they happen.<br>
<br>
2. The random cancellation<br>
<br>
As above, a job will be generating the expected output, and then
we see<br>
<blockquote>srun: forcing job termination<br>
srun: Job step aborted: Waiting up to 32 seconds for job step to
finish.<br>
slurmstepd: error: *** STEP 3531.0 ON nodename CANCELLED AT
2019-01-30T07:35:50 ***<br>
srun: error: nodename: task 250: Terminated<br>
srun: Terminating job step 3531.0<br>
</blockquote>
But this time, sacct reports<br>
<blockquote><tt> JobID Start
End ExitCode State </tt><br>
<tt>------------ ------------------- -------------------
-------- ---------- </tt><br>
<tt>3531 2019-01-30T07:21:25 2019-01-30T07:35:50
0:0 COMPLETED </tt><br>
<tt>3531.0 2019-01-30T07:21:27 2019-01-30T07:35:56
0:15 CANCELLED </tt><br>
</blockquote>
I think we've seen these cancellations pop up as soon as a minute
or two into the test run, up to perhaps 20 minutes into the run.<br>
<br>
The only thing slightly unusual in our job submissions is that we
use srun's "--immediate=120" so that the scripts can respond
appropriately if a node goes down.<br>
<br>
With SlurmctldDebug=debug2 and SlurmdDebug=debug5, there's not a
clue in the slurmctld or slurmd logs.<br>
<br>
Any thoughts on what might be happening, or what I might try next?<br>
<br>
Andy<br>
</div>
<br>
</body>
</html>