[slurm-users] [External] Autoset job TimeLimit to fit in a reservation

[Prentice Bisbal]

Florian,


Not the OP, but I have a use case where we do this. We have a code that 
uses checkpoint/restart, so for the users, being able to calculate how 
much time they have until our maintenance reservation starts allows them 
to submit restartable jobs right up to the reservation start time to 
maximize cluster usage.


For checkpoint/restart-capable jobs, on clusters where routine 
maintenance reservations are a thing, this can claim back valuable 
CPU-hours that would otherwise go to waste.


I don't know the exact code they use to check reservation time, but I 
know they do look at existing reservations and get the time when the 
next reservation starts. The first time I set maintenance reservations 
for several month's of maintenance in advance, they had a cow because 
they didn't account for the situation where there'd be more than one 
reservation in their code.

Prentice

On 3/29/21 4:09 PM, Florian Zillner wrote:
> Hi,
>
> well, I think you're putting the cart before the horse, but anyway, 
> you could write a script that extracts the next reservation and does 
> some simple math to display the time in hours or else to the user. 
> It's the users job to set the time their job needs to finish. 
> Auto-squeezing a job that takes 2 days to complete into a remaining 2 
> hour window until the reservation starts doesn't make any sense to me.
>
> # NEXTRES=$(scontrol show res | head -n1 | awk '{print $2}' | cut -f2 
> -d= | xargs -I {} date +%s --date "{}" )
> # NOW=$(date +%s)
> # echo "$(((NEXTRES - NOW) / 3600)) hours left until reservation begins"
> 178 hours left until reservation begins
>
> Cheers,
> Florian
>
>
> ------------------------------------------------------------------------
> *From:* slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf 
> of Jeremy Fix <Jeremy.Fix at centralesupelec.fr>
> *Sent:* Monday, 29 March 2021 10:48
> *To:* slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
> *Subject:* [External] [slurm-users] Autoset job TimeLimit to fit in a 
> reservation
> Hi,
>
> I'm wondering if there is any built-in option to autoset a job TimeLimit
> to fit within a defined reservation.
>
> For now, it seems to me that the timelimit must be explicitely provided,
> in a agreement with the deadline of the reservation, by a user when
> invoking the srun or sbatch command while I would find it comfortable to
> let slurm calculate the remaining time of the reservation on which
> srun/sbatch is executed and fill in the TimeLimit accordingly;
>
> Best;
>
> Jeremy.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210329/989077f0/attachment.htm>