[slurm-users] [External] Autoset job TimeLimit to fit in a reservation

[Jeremy Fix]

Hi Florian, and first of all , thanks for this single line command which
is already quite a lot; To give it a bit more sense, I will compute the
remaining time until the reservation ends, so just changing $2 in $3 in
your command

    # WEEKRESEND=$(scontrol show res | head -n1 | awk '{print $3}' | cut
    -f2 -d= | xargs -I {} date +%s --date "{}" )
    # NOW=$(date +%s)
    # echo "$(((WEEKRESEND - NOW) / 3600)) hours left until reservation
    ends"
    178 hours left until reservation ends


Trying to give some sense, here is the use case. We use slurm for
allocating resources for teaching purposes. We want to allow the
students to allocate as they wish during the week ends, starting friday
evening up to monday early morning. For that we think about creating a
reservation that is regularly scheduled every week and assigned to a
partition with part of the nodes we want the students to be able to work
with.   Then comes the reason why we need to set the timelimit.    The
partition have their timelimit and we set it to, say, 2 days and a half
for a job submitted on friday evening to be allowed to run until monday
morning. Now, suppose a user starts a job on  sunday evening without
specifying the timelimit of its jobs, the default will be the
partition's one (2 days and a half) and if there is a reservation
scheduled on monday morning, for a practical say, if I'm not wrong,
slurm will not allow the allocation because it will consider that sunday
evening + the default partition timelimit will overlap with a planned
reservation, which makes sense.   The first answer could be to ask the
user to specify the timelimit on its job so that it's allocation does
not overlap with the next reservation but we thought about automatically
setting that, hence my question.

So, in our use case, it's not a problem to kill a job by the end of the
reservation even if the job is not completed because we may need the
resources for a practical on monday morning.

Now, back to your proposal; I was thinking about putting that line in a
job prolog but .... maybe that's was you meant by "putting the cart
before the horse", I may run into troubles :     for the prolog to be
executed, the job has to the allocated but It will not with the default
timelimit of the partition because slurm may not allow it (given a
practical may be reserved on monday morning) .  Do you think there is
any other way than asking the user to set the timelimit himself in its
srun/sbatch command ?

Best;

Jeremy.

On 29/03/2021 22:09, Florian Zillner wrote:
> Hi,
>
> well, I think you're putting the cart before the horse, but anyway,
> you could write a script that extracts the next reservation and does
> some simple math to display the time in hours or else to the user.
> It's the users job to set the time their job needs to finish.
> Auto-squeezing a job that takes 2 days to complete into a remaining 2
> hour window until the reservation starts doesn't make any sense to me.
>
> # NEXTRES=$(scontrol show res | head -n1 | awk '{print $2}' | cut -f2
> -d= | xargs -I {} date +%s --date "{}" )
> # NOW=$(date +%s)
> # echo "$(((NEXTRES - NOW) / 3600)) hours left until reservation begins"
> 178 hours left until reservation begins
>
> Cheers,
> Florian
>
>
> ------------------------------------------------------------------------
> *From:* slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf
> of Jeremy Fix <Jeremy.Fix at centralesupelec.fr>
> *Sent:* Monday, 29 March 2021 10:48
> *To:* slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
> *Subject:* [External] [slurm-users] Autoset job TimeLimit to fit in a
> reservation
>  
> Hi,
>
> I'm wondering if there is any built-in option to autoset a job TimeLimit
> to fit within a defined reservation.
>
> For now, it seems to me that the timelimit must be explicitely provided,
> in a agreement with the deadline of the reservation, by a user when
> invoking the srun or sbatch command while I would find it comfortable to
> let slurm calculate the remaining time of the reservation on which
> srun/sbatch is executed and fill in the TimeLimit accordingly;
>
> Best;
>
> Jeremy.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210330/59b95c28/attachment.htm>