<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Gérard,<div class=""><br class=""></div><div class="">It is not exactly true that you have no solution to limit projects. If you implement each project as an account then you can create an account qos with the NoDecay flags.</div><div class="">This will not affect associations so priority and fair share are not impacted.</div><div class=""><br class=""></div><div class="">The way we do it is to create a qos:</div><div class=""><br class=""></div><div class="">sacctmgr -i --quiet create qos "{{ item.account }}" set flags=DenyOnLimit,NoDecay GrpTRESMin=cpu=600<br class=""></div><div class=""><br class=""></div><div class="">And then use this qos when the account (project) is created:</div><div class=""><br class=""></div><div class="">sacctmgr -i --quiet add account "{{ item.account }}" Parent="{{ item.parent }}" QOS="{{ item.account }}" Fairshare=1 Description="{{ item.description }}”</div><div class=""><br class=""></div><div class="">We even have a slurm bank implementation to play along with this technique and it has not failed us yet too much! :)</div><div class=""><br class=""></div><div class="">Hope that helps,</div><div class=""><br class=""></div><div class="">Miguel Afonso Oliveira</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 23 Jun 2022, at 14:57, <a href="mailto:gerard.gil@cines.fr" class="">gerard.gil@cines.fr</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: "trebuchet ms", sans-serif; font-size: 11pt;" class=""><div class="">Hi Ole and B/H,<br class=""><br class="">Thanks for your answers.<br class=""></div><div class=""><br data-mce-bogus="1" class=""></div><div class=""><br data-mce-bogus="1" class=""></div><div class="">You're right B/H, and as I tuned <strong class="">TRESBillingWeights</strong> option to only counts cpu, in my case : <!--StartFragment-->nb of reserved core <strong class="">= </strong> <!--StartFragment-->"TRES billing cost" <!--EndFragment--> <!--EndFragment--> </div><div class=""><br data-mce-bogus="1" class=""></div><div class="">You're right again I forgot the <strong class="">PriorityDecayHalfLife</strong> parameter which is also used by fairshare <strong class="">Multifactor Priority.</strong><!--StartFragment--><!--EndFragment--> <br class="">We use multifactor priority to manage the priority of jobs in the queue, and we set the values of <strong class="">PriorityDecayHalfLife</strong> and <strong class="">PriorityUsageResetPeriod</strong> according to these needs.<br class="">So <!--StartFragment--><em class=""><strong class="">PriorityDecayHalfLife</strong></em> <!--EndFragment-->will decay <strong class="">GrpTRESRaw</strong><!--EndFragment--> and <!--StartFragment-->GrpTRESMins <!--EndFragment--> can't be used as we want.<br class=""><br class="">Setting <!--StartFragment--><em class="">the <strong class="">NoDecay</strong> flag</em><!--EndFragment--> to a QOS could be an option but I suppose it also impact fairshare <strong class="">Multifactor Priority</strong> of all jobs using this QOS<strong class="">.</strong></div><div class=""><strong class=""><br data-mce-bogus="1" class=""></strong></div><div class="">This means I have no solution to limit a project as we want, unless schedMD changes its behavior or adds a new feature.<!--EndFragment--> <!--EndFragment--> <br class=""><br class="">Thanks a lot.<br class=""><br class="">Regards, <br class="">Gérard<span style="color: #3333ff;" class=""><br class=""></span><a href="http://www.cines.fr/" target="_blank" class=""></a><br class=""></div><div class=""><br class=""></div><hr id="zwchr" data-marker="__DIVIDER__" class=""><div data-marker="__HEADERS__" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">De: </b>"Bjørn-Helge Mevik" <<a href="mailto:b.h.mevik@usit.uio.no" class="">b.h.mevik@usit.uio.no</a>><br class=""><b class="">À: </b><a href="mailto:slurm-users@schedmd.com" class="">slurm-users@schedmd.com</a><br class=""><b class="">Envoyé: </b>Jeudi 23 Juin 2022 12:39:27<br class=""><b class="">Objet: </b>Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage<br class=""></blockquote></div><div data-marker="__QUOTED_TEXT__" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" class="">Ole.H.Nielsen@fysik.dtu.dk</a>> writes:<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Hi Bjørn-Helge,</blockquote><br class=""> <br class=""> Hello, Ole! :)<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> On 6/23/22 09:18, Bjørn-Helge Mevik wrote:<br class=""><br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Slurm the same internal variables are used for fairshare calculations as<br class=""> for GrpTRESMins (and similar), so when fair share priorities are in use,<br class=""> slurm will reduce accumulated GrpTRESMins over time. This means that it<br class=""> is impossible(*) to use GrpTRESMins limits and fairshare<br class=""> priorities at the same time.</blockquote><br class=""><br class=""> This is a surprising observation!</blockquote><br class=""> <br class=""> I discovered it quite a few years ago, when we wanted to use Slurm to<br class=""> enforce cpu hour quota limits (instead of using Maui+Gold). Can't<br class=""> remember anymore if I was surprised or just sad. :D<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> We use a 14 days HalfLife in slurm.conf:<br class=""> PriorityDecayHalfLife=14-0<br class=""><br class=""> Since our longest running jobs can run only 7 days, maybe our limits<br class=""> never get reduced as you describe?</blockquote><br class=""> <br class=""> The accumulated usage is reduced every 5 minutes (by default; see<br class=""> PriorityCalcPeriod). The reduction is done by multiplying the<br class=""> accumulated usage by a number slightly less than 1. The number is<br class=""> chosen so that the accumulated usage is reduced to 50 % after<br class=""> PriorityDecayHalfLife (given that you don't run anything more in<br class=""> between, of course). With a halflife of 14 days and the default calc<br class=""> period, that number is very close to 1 (0.9998281 if my calculations are<br class=""> correct :).<br class=""> <br class=""> Note: I read all about these details on the schedmd web pages some years<br class=""> ago. I cannot find them again (the parts about the multiplication with<br class=""> a number smaller than 1 to get the half life), so I might be wrong on<br class=""> some of the details.<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> BTW, I've written a handy script for displaying user limits in a<br class=""> readable format:<br class=""> <a href="https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits" class="">https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits</a></blockquote><br class=""> <br class=""> Nice!<br class=""> <br class=""> --<br class=""> B/H<br class=""><br class=""></blockquote></div></div></div></div></blockquote></div><br class=""></div></body></html>