<div dir="ltr"><div>Hi Paddy,</div><div><br></div><div>Thanks for the comments and suggestions!</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(7,55,99)"><font size="2">____</font></span></span><span style="color:rgb(7,55,99)"><font size="2"><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">_____</span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span></span>_____</span></span>______<br><b>Jacob D. Chappell, CSM</b><br><b>Research Computing Associate</b><br>Research Computing | Research Computing Infrastructure<br>Information Technology Services | University of Kentucky<br><a href="mailto:jacob.chappell@uky.edu" target="_blank">jacob.chappell@uky.edu</a><br></span></font></span></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 28, 2019 at 12:56 PM Paddy Doyle <<a href="mailto:paddy@tchpc.tcd.ie">paddy@tchpc.tcd.ie</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Jacob,<br>
<br>
On Tue, May 28, 2019 at 11:38:23AM -0400, Jacob Chappell wrote:<br>
<br>
> Hello all,<br>
> <br>
> Is it possible in Slurm to check RawUsage against GrpTRESMins and prevent a<br>
> job from being submitted if the RawUsage exceeds the GrpTRESMins? My center<br>
> needs this feature for detailed accounting constraints. The RawUsage is<br>
> important to us because we weight certain resource types and want to bill<br>
> people appropriately (e.g., if you use 12 GB of RAM and 1 CPU you should be<br>
> billed for 2 CPUs).<br>
<br>
That sounds like you want 'AccountingStorageEnforce=safe'; see:<br>
<br>
<a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Faccounting.html&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=NvSlTWVW1FLzuU70n2h%2BuUkynFO3cnITbx%2BXRdTIBaI%3D&reserved=0" rel="noreferrer" target="_blank">https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Faccounting.html&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=NvSlTWVW1FLzuU70n2h%2BuUkynFO3cnITbx%2BXRdTIBaI%3D&reserved=0</a><br>
<br>
And you probably want to think about PriorityDecayHalfLife and<br>
PriorityUsageResetPeriod, to control what happens after someone does hit<br>
the limit.<br>
<br>
That won't prevent the jobs from being submitted though (if I read your<br>
question correctly); it will stop them from running.<br>
<br>
We have written a little add-on 'slurm-bank' which ties together the<br>
RawUsage from sshare (in seconds) and GrpTRESMins from sacctmgr (in<br>
minutes) into a single balance statement:<br>
<br>
<a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjcftang%2Fslurm-bank&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=5T88ku58LYYJ4BK7jOzpnUVl%2Bvq3SmCjT2bLS2M0gjQ%3D&reserved=0" rel="noreferrer" target="_blank">https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjcftang%2Fslurm-bank&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=5T88ku58LYYJ4BK7jOzpnUVl%2Bvq3SmCjT2bLS2M0gjQ%3D&reserved=0</a><br>
<br>
Note that it only does simple cpu usage counting and doesn't support<br>
weights for different amounts of RAM+CPU combinations. So it may not be<br>
useful for you.<br>
<br>
Thanks,<br>
Paddy<br>
<br>
> I am thinking about implementing a custom Slurm accounting plugin, but<br>
> wanted to check to see if this feature existed already first. Or, perhaps<br>
> the feature is a work in progress and planned for a future release? Are<br>
> there any upcoming Slurm changes that might break such a plugin if a custom<br>
> implementation is necessary?<br>
> <br>
> Thanks,<br>
> __________________________________________________<br>
> *Jacob D. Chappell, CSM*<br>
> *Research Computing Associate*<br>
> Research Computing | Research Computing Infrastructure<br>
> Information Technology Services | University of Kentucky<br>
> <a href="mailto:jacob.chappell@uky.edu" target="_blank">jacob.chappell@uky.edu</a><br>
<br>
-- <br>
Paddy Doyle<br>
Trinity Centre for High Performance Computing,<br>
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.<br>
Phone: +353-1-896-3725<br>
<a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tchpc.tcd.ie%2F&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=k%2Bc9k70noPzfWANABUu8jgBDJ1Gue6qV72aFD4p8xAA%3D&reserved=0" rel="noreferrer" target="_blank">https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.tchpc.tcd.ie%2F&data=02%7C01%7Cjacob.chappell%40uky.edu%7Cc75abaa6578f450426cf08d6e38d6434%7C2b30530b69b64457b818481cb53d42ae%7C0%7C0%7C636946593746913338&sdata=k%2Bc9k70noPzfWANABUu8jgBDJ1Gue6qV72aFD4p8xAA%3D&reserved=0</a><br>
<br>
</blockquote></div>