[slurm-users] Weekend Partition

[Matthew BETTINGER]

Ok thanks for the reassurance.  I had thought about your first partition idea vs the reservation.  I was not sure if this was possible on the fly without updating the slurm.conf on each compute node without causing issues.  

On 7/23/19, 1:00 PM, "slurm-users on behalf of Daniel Letai" <slurm-users-bounces at lists.schedmd.com on behalf of dani at letai.org.il> wrote:

    I would use a partition with very low priority and preemption.
    General cluster conf:
    
    PreemptType=preempt/partition_prio
    Preemptmode=Cancel # Anything except 'Off'
    
    
    
    Partition definition:
    
    ParttionName=weekend PreemptMode=Cancel MaxTime=Unlimited PriorityTier=1 State=Down
    
    Use cron to 'scontrol update PartitionName=weekend state=up' when desired and 'scontrol update PartitionName=weekend state=down' on Sunday.
    This will not cancel the jobs on it's own, but will prevent new ones from starting.
    The preemption will kill jobs as required to allow regular jobs to run - the added value is that as long as they don't prevent other jobs from starting, those jobs can continue, and won't be killed needlessly.
    
    
    Just my 2 cents.
    
    
    The other option is to use a recurring reservation with a start and stop time frame, and force jobs to use that reservation  (possibly via qos).
    This solution might look something like:
    scontrol create reservation StartTime=00:00:01 Duration= Flags=
    For Flags you have a couple of options:
    WEEKEND
        Repeat the reservation at the same time on every weekend day (Saturday and Sunday).
    
    WEEKLY
        Repeat the reservation at the same time every week.
    So I would guess Duration=1-0 Flags=WEEKEND
    Or Duration=2-0 Flags=WEEKLY
    
    
    You will have to test to see what works best for you.
    
    
    
    HTH
    --Dani_L.
    
    
    
    On 7/23/19 7:36 PM, Matthew BETTINGER wrote:
    
    
    Hello,
    
    We run lsf and slurm here.  For LSF we have a weekend queue with no limit and jobs get killed after Sunday.  What is the best way to do something similar for slurm?  Reservation?  We would like to have any running jobs killed after Sunday if possible too.  Thanks.