<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Just to revisit this, for jobs that are queued, but prevented
from running, will have a more useful reason in 18.08, which will
address one of my issues with reservation collisions. <br>
</p>
<p><a class="moz-txt-link-freetext" href="https://bugs.schedmd.com/show_bug.cgi?id=5138">https://bugs.schedmd.com/show_bug.cgi?id=5138</a><br>
</p>
<p><a class="moz-txt-link-freetext" href="https://bugs.schedmd.com/show_bug.cgi?id=4987">https://bugs.schedmd.com/show_bug.cgi?id=4987</a></p>
<pre class="moz-signature" cols="72">Prentice Bisbal
Lead Software Engineer
Princeton Plasma Physics Laboratory
<a class="moz-txt-link-freetext" href="http://www.pppl.gov">http://www.pppl.gov</a></pre>
<div class="moz-cite-prefix">On 05/11/2018 10:36 AM, Douglas
Jacobsen wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAHaWJFEB8-MnOrvLey3YSn5NHkxQVswQbiQGP5vFmqMDpy5JYw@mail.gmail.com">
<div dir="ltr">A feature that many slurm users might like is
sbatch --time-min. Using both --time-min and --time a user can
specify the range of acceptable wall times limits. This can
make it much easier to keep jobs running right up to the
maintenance reservation. e.g.:
<div><br>
</div>
<div>sbatch --time-min=30:00 --time=48:00:00 script.sh</div>
<div><br>
</div>
<div>would allow the job to schedule for any time-slot between
30 minutes and 2 days in length. If the user has some
mechanism for job chaining or similar, this can allow them to
make the most of backfill opportunities.</div>
<div><br>
</div>
<div>-Doug</div>
<div><br clear="all">
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div><font size="1" face="courier new, monospace">----</font></div>
<div><font face="arial, helvetica, sans-serif"><font
size="2">Doug Jacobsen, Ph.D.</font><br>
</font></div>
<div><font size="1" face="arial, helvetica,
sans-serif">NERSC Computer Systems Engineer</font></div>
<div><font size="1"><font face="arial, helvetica,
sans-serif"><a href="http://www.nersc.gov"
target="_blank" moz-do-not-send="true">National
Energy Research Scientific Computing Center</a></font><br>
</font></div>
<div><font size="1"><a
href="mailto:dmjacobsen@lbl.gov"
style="font-family:arial,helvetica,sans-serif"
target="_blank" moz-do-not-send="true">dmjacobsen@lbl.gov</a><span
style="font-family:arial,helvetica,sans-serif"></span></font><br>
</div>
<div><font size="1"><br>
</font></div>
<div><font size="1"><font face="courier new,
monospace"><span
style="color:rgb(136,136,136)">-------------
__o</span><br style="color:rgb(136,136,136)">
<span style="color:rgb(136,136,136)">----------
_ '\<,_</span><br
style="color:rgb(136,136,136)">
<span style="color:rgb(136,136,136)">----------(_)/
(_)__________________________</span></font><br>
</font></div>
<div style="font-family:arial;font-size:small"><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, May 11, 2018 at 7:27 AM Paul Edmon <<a
href="mailto:pedmon@cfa.harvard.edu" moz-do-not-send="true">pedmon@cfa.harvard.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">In the past
we used the LUA job submit plugin to block jobs that would <br>
intersect maintenance reservations. I would look at that.<br>
<br>
-Paul Edmon-<br>
<br>
<br>
On 05/11/2018 08:19 AM, Bill Wichser wrote:<br>
> The problem is that reservations can be in there yet have
no effect on <br>
> the submitted job if they would run before the
reservation takes <br>
> place. One can pull the starting time simply using
something like this<br>
><br>
> scontrol show res -o | awk '{print $2}'<br>
><br>
> with output<br>
><br>
> StartTime=2018-06-12T06:00:00<br>
> StartTime=2018-06-12T06:00:00<br>
><br>
> You'd need more code around that, obviously, to determine
if this <br>
> starttime might hold up the job.<br>
><br>
> Bill<br>
><br>
><br>
> On 05/10/2018 04:23 PM, Prentice Bisbal wrote:<br>
>> Dear Slurm Users,<br>
>><br>
>> We've started using maintenance reservations. As you
would expect, <br>
>> this caused some confusion for users who were
wondering why their <br>
>> jobs were queuing up and not running. Some of my
users provide a <br>
>> public service of sorts that automatically submits
jobs to our <br>
>> cluster. They would like to have their submission
framework <br>
>> automatically detect if there's a reservation that
may interfere with <br>
>> their jobs, and act accordingly.<br>
>><br>
>> What is the best way to do this? Typically, in my
shell scripts, I <br>
>> have some command that tests something, and then
check exit code <br>
>> returned by the command. For example to check if my
name is in file <br>
>> 'foo.txt', I'd do something like this:<br>
>><br>
>> grep -iq prentice foo.txt<br>
>> retval=$?<br>
>> if [ $retval -eq 0 ]; then<br>
>> echo "Prentice found"<br>
>> else<br>
>> echo "Prentice not found"<br>
>> fi<br>
>> unset retval<br>
>><br>
>> Or something like that. I was also thinking this
might work, too:<br>
>><br>
>> num_res=$(scontrol -o show res | wc -l)<br>
>> if [ $num_res -eq 0 ]; then<br>
>> echo "No reservations found"<br>
>> else<br>
>> echo "$num_res reservation(s) found"<br>
>> fi<br>
>><br>
>> Are there any better or other ways that you would
recommend? Also, if <br>
>> there's more than one, is are they listed in any kind
of order in the <br>
>> scontrol or sinfo output (soonest first, soonest
last, etc.)? From <br>
>> the man page, it looks like 'scontrol show
reservation' doesn't <br>
>> provide any sorting.<br>
>><br>
>> Prentice<br>
>><br>
>><br>
>><br>
>><br>
>><br>
><br>
<br>
<br>
</blockquote>
</div>
</blockquote>
<br>
</body>
</html>