[slurm-users] Preempt jobs to stay within account TRES limits?

[Matthew R. Baney]

Hello,

I have noticed that jobs submitted to non-preemptable partitions
(PreemptType = preempt/partition_prio and PreemptMode = REQUEUE) under
accounts with GrpTRES limits will become pending with AssocGrpGRES as the
reason when the account is up against the relevant limit, even when there
are other running jobs on preemptable partitions under the same account and
when the pending jobs have higher priority. The GRES in consideration are
GPUs.

It seems like the scheduler is checking to see if the pending jobs are
within the GRES limit for the account before considering if any of the
other jobs in the account are running on preemptable partitions. In some
specific observed cases, even preempting a single job running in a
preemptable partition would allow the non-preemptable partition job to run
(based on GRES freed up from preemption).

Is it possible to reverse the order in which these checks are evaluated?

Best,
Matthew

-- 
Matthew Baney
UMIACS Technical Staff
mbaney at umd.edu | (301) 405-6756
University of Maryland Institute for Advanced Computer Studies
3154 Brendan Iribe Center
8125 Paint Branch Dr.
College Park, MD 20742
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20221021/0a0bf002/attachment.htm>