[slurm-users] Preempt jobs to stay within account TRES limits?

[Steven Dick]

QOS Group TRES limits apply to associations.
If I recall correctly, an association is a (user,account,partition,cluster)

On Fri, Oct 21, 2022 at 9:46 AM Matthew R. Baney <mbaney at umd.edu> wrote:
>
> Hello,
>
> I have noticed that jobs submitted to non-preemptable partitions (PreemptType = preempt/partition_prio and PreemptMode = REQUEUE) under accounts with GrpTRES limits will become pending with AssocGrpGRES as the reason when the account is up against the relevant limit, even when there are other running jobs on preemptable partitions under the same account and when the pending jobs have higher priority. The GRES in consideration are GPUs.
>
> It seems like the scheduler is checking to see if the pending jobs are within the GRES limit for the account before considering if any of the other jobs in the account are running on preemptable partitions. In some specific observed cases, even preempting a single job running in a preemptable partition would allow the non-preemptable partition job to run (based on GRES freed up from preemption).
>
> Is it possible to reverse the order in which these checks are evaluated?
>
> Best,
> Matthew
>
> --
> Matthew Baney
> UMIACS Technical Staff
> mbaney at umd.edu | (301) 405-6756
> University of Maryland Institute for Advanced Computer Studies
> 3154 Brendan Iribe Center
> 8125 Paint Branch Dr.
> College Park, MD 20742