[slurm-users] Weekend Partition
matthew.bettinger at external.total.com
Tue Jul 23 18:06:50 UTC 2019
Ok thanks for the reassurance. I had thought about your first partition idea vs the reservation. I was not sure if this was possible on the fly without updating the slurm.conf on each compute node without causing issues.
On 7/23/19, 1:00 PM, "slurm-users on behalf of Daniel Letai" <slurm-users-bounces at lists.schedmd.com on behalf of dani at letai.org.il> wrote:
I would use a partition with very low priority and preemption.
General cluster conf:
Preemptmode=Cancel # Anything except 'Off'
ParttionName=weekend PreemptMode=Cancel MaxTime=Unlimited PriorityTier=1 State=Down
Use cron to 'scontrol update PartitionName=weekend state=up' when desired and 'scontrol update PartitionName=weekend state=down' on Sunday.
This will not cancel the jobs on it's own, but will prevent new ones from starting.
The preemption will kill jobs as required to allow regular jobs to run - the added value is that as long as they don't prevent other jobs from starting, those jobs can continue, and won't be killed needlessly.
Just my 2 cents.
The other option is to use a recurring reservation with a start and stop time frame, and force jobs to use that reservation (possibly via qos).
This solution might look something like:
scontrol create reservation StartTime=00:00:01 Duration= Flags=
For Flags you have a couple of options:
Repeat the reservation at the same time on every weekend day (Saturday and Sunday).
Repeat the reservation at the same time every week.
So I would guess Duration=1-0 Flags=WEEKEND
Or Duration=2-0 Flags=WEEKLY
You will have to test to see what works best for you.
On 7/23/19 7:36 PM, Matthew BETTINGER wrote:
We run lsf and slurm here. For LSF we have a weekend queue with no limit and jobs get killed after Sunday. What is the best way to do something similar for slurm? Reservation? We would like to have any running jobs killed after Sunday if possible too. Thanks.
More information about the slurm-users