[slurm-users] Checking RawUsage against GrpTRESMins

[Jacob Chappell]

Hi Paddy,

Thanks for the comments and suggestions!

__________________________________________________
*Jacob D. Chappell, CSM*
*Research Computing Associate*
Research Computing | Research Computing Infrastructure
Information Technology Services | University of Kentucky
jacob.chappell at uky.edu


On Tue, May 28, 2019 at 12:56 PM Paddy Doyle <paddy at tchpc.tcd.ie> wrote:

> Hi Jacob,
>
> On Tue, May 28, 2019 at 11:38:23AM -0400, Jacob Chappell wrote:
>
> > Hello all,
> >
> > Is it possible in Slurm to check RawUsage against GrpTRESMins and
> prevent a
> > job from being submitted if the RawUsage exceeds the GrpTRESMins? My
> center
> > needs this feature for detailed accounting constraints. The RawUsage is
> > important to us because we weight certain resource types and want to bill
> > people appropriately (e.g., if you use 12 GB of RAM and 1 CPU you should
> be
> > billed for 2 CPUs).
>
> That sounds like you want 'AccountingStorageEnforce=safe'; see:
>
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Faccounting.html&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=NvSlTWVW1FLzuU70n2h%2BuUkynFO3cnITbx%2BXRdTIBaI%3D&reserved=0
>
> And you probably want to think about PriorityDecayHalfLife and
> PriorityUsageResetPeriod, to control what happens after someone does hit
> the limit.
>
> That won't prevent the jobs from being submitted though (if I read your
> question correctly); it will stop them from running.
>
> We have written a little add-on 'slurm-bank' which ties together the
> RawUsage from sshare (in seconds) and GrpTRESMins from sacctmgr (in
> minutes) into a single balance statement:
>
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjcftang%2Fslurm-bank&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=5T88ku58LYYJ4BK7jOzpnUVl%2Bvq3SmCjT2bLS2M0gjQ%3D&reserved=0
>
> Note that it only does simple cpu usage counting and doesn't support
> weights for different amounts of RAM+CPU combinations. So it may not be
> useful for you.
>
> Thanks,
> Paddy
>
> > I am thinking about implementing a custom Slurm accounting plugin, but
> > wanted to check to see if this feature existed already first. Or, perhaps
> > the feature is a work in progress and planned for a future release? Are
> > there any upcoming Slurm changes that might break such a plugin if a
> custom
> > implementation is necessary?
> >
> > Thanks,
> > __________________________________________________
> > *Jacob D. Chappell, CSM*
> > *Research Computing Associate*
> > Research Computing | Research Computing Infrastructure
> > Information Technology Services | University of Kentucky
> > jacob.chappell at uky.edu
>
> --
> Paddy Doyle
> Trinity Centre for High Performance Computing,
> Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
> Phone: +353-1-896-3725
>
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tchpc.tcd.ie%2F&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=k%2Bc9k70noPzfWANABUu8jgBDJ1Gue6qV72aFD4p8xAA%3D&reserved=0
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190528/c25e6af7/attachment-0001.html>