[slurm-users] Checking RawUsage against GrpTRESMins

[Paddy Doyle]

Hi Jacob,

On Tue, May 28, 2019 at 11:38:23AM -0400, Jacob Chappell wrote:

> Hello all,
> 
> Is it possible in Slurm to check RawUsage against GrpTRESMins and prevent a
> job from being submitted if the RawUsage exceeds the GrpTRESMins? My center
> needs this feature for detailed accounting constraints. The RawUsage is
> important to us because we weight certain resource types and want to bill
> people appropriately (e.g., if you use 12 GB of RAM and 1 CPU you should be
> billed for 2 CPUs).

That sounds like you want 'AccountingStorageEnforce=safe'; see:

https://slurm.schedmd.com/accounting.html

And you probably want to think about PriorityDecayHalfLife and
PriorityUsageResetPeriod, to control what happens after someone does hit
the limit.

That won't prevent the jobs from being submitted though (if I read your
question correctly); it will stop them from running.

We have written a little add-on 'slurm-bank' which ties together the
RawUsage from sshare (in seconds) and GrpTRESMins from sacctmgr (in
minutes) into a single balance statement:

https://github.com/jcftang/slurm-bank

Note that it only does simple cpu usage counting and doesn't support
weights for different amounts of RAM+CPU combinations. So it may not be
useful for you.

Thanks,
Paddy

> I am thinking about implementing a custom Slurm accounting plugin, but
> wanted to check to see if this feature existed already first. Or, perhaps
> the feature is a work in progress and planned for a future release? Are
> there any upcoming Slurm changes that might break such a plugin if a custom
> implementation is necessary?
> 
> Thanks,
> __________________________________________________
> *Jacob D. Chappell, CSM*
> *Research Computing Associate*
> Research Computing | Research Computing Infrastructure
> Information Technology Services | University of Kentucky
> jacob.chappell at uky.edu

-- 
Paddy Doyle
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
Phone: +353-1-896-3725
http://www.tchpc.tcd.ie/