<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>We recently upgraded from Slurm 19.05.8 to 20.11.3. In our
configuration, we have an interruptible partition named
'interruptible' for long-running, low-priority jobs that use
checkpoint/restart. Jobs that are preempted would be killed and
requeued rather than suspended. This configuration has been
working without issue for 2+ years without issue. <br>
</p>
<p>After the upgrade, this has stopped working. Preempted jobs are
killed and not requeued. My slurm.conf file is configured to
requeue preempted jobs:</p>
<p>$ grep -i requeue /etc/slurm/slurm.conf <br>
#JobRequeue=1<br>
PreemptMode=Requeue<br>
</p>
<p>And the user's sbatch script included the --requeue option. <br>
</p>
<p>The user reports the err output from his preempted jobs now says</p>
<p><span class="tabs2_section tabs2_section_1 tabs2_section1
tab_section" data-header-only="false"
id="section_tab.991f88d20a00064127420bc37824d385"
data-section-id="991f88d20a00064127420bc37824d385"
style="display: block;" role="tabpanel" aria-hidden="false"
aria-labelledby="section_tab.991f88d20a00064127420bc37824d385"><span
id="section-991f88d20a00064127420bc37824d385"
data-header-only="false" class="section sn-stream-section"><span
class="sn-widget-textblock-body
sn-widget-textblock-body_formatted">slurmstepd: error: ***
STEP 1075117.0 ON greene002 CANCELLED AT 2021-02-25T16:07:48
***</span></span></span></p>
<p><span class="tabs2_section tabs2_section_1 tabs2_section1
tab_section" data-header-only="false"
id="section_tab.991f88d20a00064127420bc37824d385"
data-section-id="991f88d20a00064127420bc37824d385"
style="display: block;" role="tabpanel" aria-hidden="false"
aria-labelledby="section_tab.991f88d20a00064127420bc37824d385"><span
id="section-991f88d20a00064127420bc37824d385"
data-header-only="false" class="section sn-stream-section"><span
class="sn-widget-textblock-body
sn-widget-textblock-body_formatted">And in the past it would
see PREEMPTED instead of cancelled. <br>
</span></span></span></p>
<p><span class="tabs2_section tabs2_section_1 tabs2_section1
tab_section" data-header-only="false"
id="section_tab.991f88d20a00064127420bc37824d385"
data-section-id="991f88d20a00064127420bc37824d385"
style="display: block;" role="tabpanel" aria-hidden="false"
aria-labelledby="section_tab.991f88d20a00064127420bc37824d385"><span
id="section-991f88d20a00064127420bc37824d385"
data-header-only="false" class="section sn-stream-section"><span
class="sn-widget-textblock-body
sn-widget-textblock-body_formatted">Any ideas what would
cause this? I've reported this to Slurm support, and haven't
gotten anything back yet, so I figured I'd ask here, too. If
this is a bug, I can't be the only one who has experienced
this. <br>
</span></span></span></p>
<pre class="moz-signature" cols="72">--
Prentice </pre>
</body>
</html>