[slurm-users] tie a reservation to a QoS?

[Bill Wichser]

One thing we changed years ago was to think about things differently. 
While researchers are in fact buying nodes for the cluster, it's rarely 
the case that they get any rights to "their" nodes.  Instead they are 
buying CPU time in an equivalent way but it gets averaged over 30 days.

We provide reports, explain how fairshare works and how their particular 
value was chosen, give them all the ways that we as sysadmins look at 
the data, and for the most part have accepted that buy-in means CPU 
hours/month rather than a certain piece of hardware.

Now that journey was not easy.  And there is a discussion with every new 
researcher who wants a piece of the cluster.  But it works for us.  But 
there is a big caveat.  There needs to be at least a 10% share of the 
cluster which is owned by a public portion.

This gives some leeway into scheduling and allows others who have not 
contributed to have access to the same resources albeit at a much lower 
priority.  And that 10% for us has become more like 25% or even more as 
we have a large and ever growing base of users who have not contributed.

This different way of thinking has made dedicated partitions and QOSes 
something we have not had to deal with as CPU time per 30 day sliding 
window has been accepted, can be quantitatively shown, and just is a 
much easier way to schedule when ALL resources can be used.

Bill

On 10/28/19 11:11 AM, Tina Friedrich wrote:
> Hello,
> 
> is there a possibility to tie a reservation to a QoS (instead of an
> account or user), or enforce a QoS for jobs submitted into a reservation?
> 
> The problem I'm trying to solve is - some of our resources are bought on
> a co-investment basis. As part of that, the 'owning' group can get very
> high scheduling priority (via a QoS) on an equivalent amount of
> resource. Additionally, they have a number of reservations for 'their'
> nodes they can request per year. However, that lends itself to gaming
> the system - they can now submit jobs into the reservation with 'normal'
> priority, and then run jobs on the rest of the cluster using the higher
> priority - really not the plan.
> 
> Basically, I need a way to ensure that - even when a reservation is in
> place - those groups 'use up' their priority resources first & then all
> other jobs they submit are run with 'lower' priority.
> 
> I'm currently dealing with it by modifying the QoS every time a
> reservation is created. But that isn't really sustainable on an ongoing
> basis - this isn't a one-off for one group, it's part of our operations
> model, and there's a (growing) number of them.
> 
> One (easy) way I can see is if I had a way to ensure you can not use a
> reservation without using the resp. priority QoS - however, from my
> reading of the docs there's no way to do that. (As only the one account
> has access to the QoS, being able to tie a reservation to a QoS would
> sort of solve my problem :) ).
> 
> Any ideas? The only thing I can come up involves a lot of scripting, and
> it would certainly not be a bit error prone (and not the most flexible).
> 
> Tina
>