[slurm-users] Preemption within same QOS

[Relu Patrascu]

We're having a bit of a problem setting up slurm to achieve this:

1. Two QOSs, 'high' and 'normal'.
2. Preemption type: requeue.
3. Any job has a guarantee of running 60 minutes before being preempted.
4. Any job submitted with --qos=high can preempt jobs with --qos=normal if
no resources available and all jobs with --qos=normal have been running for
at least 60 minutes.
5. Job A can preempt job B in the same QOS if all the resources are
allocated and if job B has run for at least 60 minutes, and if job B has
lower priority than job A. If job A preempts Job B then B is requeued.

We already set this up, but slurm does not allow loops in QOS preemption.
That is, QOS 'normal' cannot preempt QOS 'normal'. Is there a way to
achieve what we want without having to change the source code? We actually
did try modifying the source code to allow preemption within the same QOS,
but what we're observing is a job with a lower priority can preempt a job
with a higher priority, after the 60 minutes grace time that we have set.
We use the builtin scheduler:

SchedulerType           = sched/builtin
SchedulerParameters     = preempt_strict_order,preempt_reorder_count=5

sacctmgr show qos
format=Name,Priority,Preempt%20,PreemptExemptTime,PreemptMode
      Name   Priority              Preempt   PreemptExemptTime PreemptMode

---------- ---------- -------------------- ------------------- -----------
-------------
    normal          1               normal            01:00:00     requeue

      high       1000          high,normal            01:00:00     requeue


Any ideas how we could go about to achieve what we want?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200122/02ed1fc2/attachment.htm>