[slurm-users] [External] Autoset job TimeLimit to fit in a reservation

[Renfro, Michael]

I'd probably write a shell function that would calculate the time required, and add it as a command-line parameter to sbatch. We do a similar thing for easier interactive shells in our /etc/profile.d folder on the login node:

function hpcshell() {
  srun --partition=interactive $@ --pty bash -i
}

So something like (untested):

weekendjob () {
  WEEKRESEND=$(scontrol show res | head -n1 | awk '{print $3}' | cut -f2 -d= | xargs -I {} date +%s --date "{}" )
  NOW=$(date +%s)
  HOURS=$(((WEEKRESEND - NOW) / 3600))
  sbatch --time=${HOURS}:00:00 $@
}

and:

  weekendjob myjob.sh

could work, with the possible downside that anyone using this early in the reservation period would potentially reserve resources for the entire weekend.

From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Jeremy Fix <Jeremy.Fix at centralesupelec.fr>
Date: Tuesday, March 30, 2021 at 2:24 AM
To: Florian Zillner <fzillner at lenovo.com>, slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] [External] Autoset job TimeLimit to fit in a reservation

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

________________________________
Hi Florian, and first of all , thanks for this single line command which is already quite a lot; To give it a bit more sense, I will compute the remaining time until the reservation ends, so just changing $2 in $3 in your command

# WEEKRESEND=$(scontrol show res | head -n1 | awk '{print $3}' | cut -f2 -d= | xargs -I {} date +%s --date "{}" )
# NOW=$(date +%s)
# echo "$(((WEEKRESEND - NOW) / 3600)) hours left until reservation ends"
178 hours left until reservation ends

Trying to give some sense, here is the use case. We use slurm for allocating resources for teaching purposes. We want to allow the students to allocate as they wish during the week ends, starting friday evening up to monday early morning. For that we think about creating a reservation that is regularly scheduled every week and assigned to a partition with part of the nodes we want the students to be able to work with.   Then comes the reason why we need to set the timelimit.    The partition have their timelimit and we set it to, say, 2 days and a half for a job submitted on friday evening to be allowed to run until monday morning. Now, suppose a user starts a job on  sunday evening without specifying the timelimit of its jobs, the default will be the partition's one (2 days and a half) and if there is a reservation scheduled on monday morning, for a practical say, if I'm not wrong, slurm will not allow the allocation because it will consider that sunday evening + the default partition timelimit will overlap with a planned reservation, which makes sense.   The first answer could be to ask the user to specify the timelimit on its job so that it's allocation does not overlap with the next reservation but we thought about automatically setting that, hence my question.

So, in our use case, it's not a problem to kill a job by the end of the reservation even if the job is not completed because we may need the resources for a practical on monday morning.

Now, back to your proposal; I was thinking about putting that line in a job prolog but .... maybe that's was you meant by "putting the cart before the horse", I may run into troubles :     for the prolog to be executed, the job has to the allocated but It will not with the default timelimit of the partition because slurm may not allow it (given a practical may be reserved on monday morning) .  Do you think there is any other way than asking the user to set the timelimit himself in its srun/sbatch command ?

Best;

Jeremy.

On 29/03/2021 22:09, Florian Zillner wrote:
Hi,

well, I think you're putting the cart before the horse, but anyway, you could write a script that extracts the next reservation and does some simple math to display the time in hours or else to the user. It's the users job to set the time their job needs to finish. Auto-squeezing a job that takes 2 days to complete into a remaining 2 hour window until the reservation starts doesn't make any sense to me.

# NEXTRES=$(scontrol show res | head -n1 | awk '{print $2}' | cut -f2 -d= | xargs -I {} date +%s --date "{}" )
# NOW=$(date +%s)
# echo "$(((NEXTRES - NOW) / 3600)) hours left until reservation begins"
178 hours left until reservation begins

Cheers,
Florian


________________________________
From: slurm-users <slurm-users-bounces at lists.schedmd.com><mailto:slurm-users-bounces at lists.schedmd.com> on behalf of Jeremy Fix <Jeremy.Fix at centralesupelec.fr><mailto:Jeremy.Fix at centralesupelec.fr>
Sent: Monday, 29 March 2021 10:48
To: slurm-users at lists.schedmd.com<mailto:slurm-users at lists.schedmd.com> <slurm-users at lists.schedmd.com><mailto:slurm-users at lists.schedmd.com>
Subject: [External] [slurm-users] Autoset job TimeLimit to fit in a reservation

Hi,

I'm wondering if there is any built-in option to autoset a job TimeLimit
to fit within a defined reservation.

For now, it seems to me that the timelimit must be explicitely provided,
in a agreement with the deadline of the reservation, by a user when
invoking the srun or sbatch command while I would find it comfortable to
let slurm calculate the remaining time of the reservation on which
srun/sbatch is executed and fill in the TimeLimit accordingly;

Best;

Jeremy.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210330/561a10ec/attachment.htm>