<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks - that's an awesome, yet horrible, hack :)<div class=""><br class=""></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Noam<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 8, 2018, at 3:26 AM, Josep Manel Andrés Moscardó <<a href="mailto:josep.moscardo@embl.de" class="">josep.moscardo@embl.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi,<br class="">Somebody else gave me this piece of code (I hope he doesn't mind me sharing it :) , at least it is how they do it:<br class=""><br class="">#!/bin/bash<br class=""><br class="">#SBATCH --signal=B:USR1@300 #<------ This will make Slurm send signal USR1 to the bash process 300 seconds before the time limit<br class="">#SBATCH -t 00:06:00<br class=""><br class="">resubmit() {<br class="">echo "It's time to resubmit"; # <----- Run whatever is necessary. Shutdown the DDBB, resubmit the job using the checkpointing files...<br class="">}<br class=""><br class="">trap "resubmit" USR1<br class=""><br class="">sleep 1000 & # <----- Your program goes here. It's important to run on the background otherwise bash will not process the signal until this command finishes<br class=""><br class="">wait # <---- wait until all the background processes are finished. If a signal is received this will stop, process the signal and finish the script.<br class=""><br class=""><br class="">On 7/11/18 21:16, Noam Bernstein wrote:<br class=""><blockquote type="cite" class="">Hi slurm users - I’ve been looking through the slurm prolog/epilog manuals, but haven’t been able to figure out if there’s a way to get an epilog script (requested by the user) to run when a job is killed for running out of time, and have the epilog script be able to detect that (through an env variable, for example). Is this possible?<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>thanks,<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span><span class="Apple-tab-span" style="white-space:pre"> </span>Noam<br class=""></blockquote><br class="">-- <br class="">Josep Manel Andrés Moscardó<br class="">Systems Engineer, IT Operations<br class="">EMBL Heidelberg<br class="">T +49 6221 387-8394<br class=""><br class=""></div></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<p class="">
<font face="Arial" class=""></font></p><div style="color: rgb(0, 0, 100); font-size: 14px" class=""><font face="Arial" class="">
<div style=" text-align: left; display: inline-block; width: 1px; margin-right:0px" class=""></div><!--
--><div style=" text-align: center; display: inline-block; width: 100px" ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="">____________</div><!--
--><div style=" text-align: right; display: inline-block; width: 1px; margin-left: -3px" class=""></div><!--
--><div class=""></div>
<div style=" text-align: left; display: inline-block; width: 1px; margin-right:0px" class="">|</div><!--
--><div style=" text-align: center; display: inline-block; width: 100px" ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class=""></div><!--
--><div style=" text-align: right; display: inline-block; width: 1px; margin-left: -3px" class="">|</div><!--
--><div class=""></div>
<div style=" text-align: left; display: inline-block; width: 1px; margin-right:0px" class="">|</div><!--
--><div style=" text-align: center; display: inline-block; width: 100px" ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class=""><b class="">U.S. NAVAL</b></div><!--
--><div style=" text-align: right; display: inline-block; width: 1px; margin-left: -3px" class="">|</div><!--
--><div class=""></div>
<div style=" text-align: left; display: inline-block; width: 1px; margin-right:0px" class="">|</div><!--
--><div style=" text-align: center; display: inline-block; width: 100px" ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class="">_<b class="">RESEARCH</b>_</div><!--
--><div style=" text-align: right; display: inline-block; width: 1px; margin-left: -3px" class="">|</div><!--
--><div class=""></div>
<div style=" text-align: left; display: inline-block; width: 1px; margin-right:0px" class=""></div><!--
--><div style=" text-align: center; display: inline-block; width: 100px" ;="" margin-left:="" 0px;="" margin-right:="" 0px"="" class=""><span style="font-size: 12px" class="">LABORATORY</span></div><!--
--><div style=" text-align: right; display: inline-block; width: 1px; margin-left: -3px" class=""></div><!--
--><div class=""></div>
</font></div>
<p class=""></p>
<div class="">Noam Bernstein, Ph.D.</div></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Center for Materials Physics and Technology</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">U.S. Naval Research Laboratory</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">T +1 202 404 8628 F +1 202 404 7546</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://www.nrl.navy.mil" class="">https://www.nrl.navy.mil</a></div>
</div>
<br class=""></div></body></html>