[slurm-users] Preemption within same QOS

[Relu Patrascu]

We received no replies, so we solved the problem in house by writing a
simple plugin based on the qos priority plugin.

On Wed, Jan 22, 2020 at 2:50 PM Relu Patrascu <relu at vectorinstitute.ai>
wrote:

> We're having a bit of a problem setting up slurm to achieve this:
>
> 1. Two QOSs, 'high' and 'normal'.
> 2. Preemption type: requeue.
> 3. Any job has a guarantee of running 60 minutes before being preempted.
> 4. Any job submitted with --qos=high can preempt jobs with --qos=normal if
> no resources available and all jobs with --qos=normal have been running for
> at least 60 minutes.
> 5. Job A can preempt job B in the same QOS if all the resources are
> allocated and if job B has run for at least 60 minutes, and if job B has
> lower priority than job A. If job A preempts Job B then B is requeued.
>
> We already set this up, but slurm does not allow loops in QOS preemption.
> That is, QOS 'normal' cannot preempt QOS 'normal'. Is there a way to
> achieve what we want without having to change the source code? We actually
> did try modifying the source code to allow preemption within the same QOS,
> but what we're observing is a job with a lower priority can preempt a job
> with a higher priority, after the 60 minutes grace time that we have set.
> We use the builtin scheduler:
>
> SchedulerType           = sched/builtin
> SchedulerParameters     = preempt_strict_order,preempt_reorder_count=5
>
> sacctmgr show qos
> format=Name,Priority,Preempt%20,PreemptExemptTime,PreemptMode
>       Name   Priority              Preempt   PreemptExemptTime
> PreemptMode
> ---------- ---------- -------------------- ------------------- -----------
> -------------
>     normal          1               normal            01:00:00     requeue
>
>       high       1000          high,normal            01:00:00
> requeue
>
> Any ideas how we could go about to achieve what we want?
>
>
>

-- 

+1-647-680-7564
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200309/a66250f9/attachment.htm>