<html><body><div style="font-family: trebuchet ms,sans-serif; font-size: 11pt; color: #000000"><div>Hi Miguel,<br></div><div><br></div><div>It sounds good !<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>But does it mean you have to request this "NoDecay" QOS to benefit the fairshare priority ?<br></div><div><br data-mce-bogus="1"></div><div>Does this also mean that if all the QOS we use are created with NoDecay, we can take advantage of the FairShare priority and NoDecay for all jobs to use the GrpTRESMins limit?<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks <br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div data-marker="__SIG_PRE__"><div><span style="color: #3333ff;"><span style="color: #000000;">Regards,</span></span></div><div><span style="color: #3333ff;"><span style="color: #000000;">Gérard </span><br></span><br></div></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>De: </b>"Miguel Oliveira" <miguel.oliveira@uc.pt><br><b>À: </b>"Slurm-users" <slurm-users@lists.schedmd.com><br><b>Cc: </b>"slurm-users" <slurm-users@schedmd.com><br><b>Envoyé: </b>Jeudi 23 Juin 2022 18:42:28<br><b>Objet: </b>Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage<br></blockquote></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">Hi Gérard,<div class=""><br class=""></div><div class="">It is not exactly true that you have no solution to limit projects. If you implement each project as an account then you can create an account qos with the NoDecay flags.</div><div class="">This will not affect associations so priority and fair share are not impacted.</div><div class=""><br class=""></div><div class="">The way we do it is to create a qos:</div><div class=""><br class=""></div><div class="">sacctmgr -i --quiet create qos "{{ item.account }}" set flags=DenyOnLimit,NoDecay GrpTRESMin=cpu=600<br class=""></div><div class=""><br class=""></div><div class="">And then use this qos when the account (project) is created:</div><div class=""><br class=""></div><div class="">sacctmgr -i --quiet add account "{{ item.account }}" Parent="{{ item.parent }}" QOS="{{ item.account }}" Fairshare=1 Description="{{ item.description }}”</div><div class=""><br class=""></div><div class="">We even have a slurm bank implementation to play along with this technique and it has not failed us yet too much! :)</div><div class=""><br class=""></div><div class="">Hope that helps,</div><div class=""><br class=""></div><div class="">Miguel Afonso Oliveira</div><div class=""><br class=""><div><br class=""><blockquote class=""><div class="">On 23 Jun 2022, at 14:57, <a href="mailto:gerard.gil@cines.fr" class="" target="_blank">gerard.gil@cines.fr</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: "trebuchet ms", sans-serif; font-size: 11pt;" class=""><div class="">Hi Ole and B/H,<br class=""><br class="">Thanks for your answers.<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">You're right B/H, and as I tuned <strong class="">TRESBillingWeights</strong> option to only counts cpu, in my case : nb of reserved core <strong class="">= </strong> "TRES billing cost" </div><div class=""><br class=""></div><div class="">You're right again I forgot the <strong class="">PriorityDecayHalfLife</strong> parameter which is also used by fairshare <strong class="">Multifactor Priority.</strong> <br class="">We use multifactor priority to manage the priority of jobs in the queue, and we set the values of <strong class="">PriorityDecayHalfLife</strong> and <strong class="">PriorityUsageResetPeriod</strong> according to these needs.<br class="">So <em class=""><strong class="">PriorityDecayHalfLife</strong></em> will decay <strong class="">GrpTRESRaw</strong> and GrpTRESMins can't be used as we want.<br class=""><br class="">Setting <em class="">the <strong class="">NoDecay</strong> flag</em> to a QOS could be an option but I suppose it also impact fairshare <strong class="">Multifactor Priority</strong> of all jobs using this QOS<strong class="">.</strong></div><div class=""><strong class=""><br class=""></strong></div><div class="">This means I have no solution to limit a project as we want, unless schedMD changes its behavior or adds a new feature. <br class=""><br class="">Thanks a lot.<br class=""><br class="">Regards, <br class="">Gérard<span style="color: #3333ff;" class=""><br class=""></span><a href="http://www.cines.fr/" target="_blank" class=""></a><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">De: </b>"Bjørn-Helge Mevik" <<a href="mailto:b.h.mevik@usit.uio.no" class="" target="_blank">b.h.mevik@usit.uio.no</a>><br class=""><b class="">À: </b><a href="mailto:slurm-users@schedmd.com" class="" target="_blank">slurm-users@schedmd.com</a><br class=""><b class="">Envoyé: </b>Jeudi 23 Juin 2022 12:39:27<br class=""><b class="">Objet: </b>Re: [slurm-users] GrpTRESMins and GrpTRESRaw usage<br class=""></blockquote></div><div class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Ole Holm Nielsen <<a href="mailto:Ole.H.Nielsen@fysik.dtu.dk" class="" target="_blank">Ole.H.Nielsen@fysik.dtu.dk</a>> writes:<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Hi Bjørn-Helge,</blockquote><br class=""> <br class=""> Hello, Ole! :)<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> On 6/23/22 09:18, Bjørn-Helge Mevik wrote:<br class=""><br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> Slurm the same internal variables are used for fairshare calculations as<br class=""> for GrpTRESMins (and similar), so when fair share priorities are in use,<br class=""> slurm will reduce accumulated GrpTRESMins over time. This means that it<br class=""> is impossible(*) to use GrpTRESMins limits and fairshare<br class=""> priorities at the same time.</blockquote><br class=""><br class=""> This is a surprising observation!</blockquote><br class=""> <br class=""> I discovered it quite a few years ago, when we wanted to use Slurm to<br class=""> enforce cpu hour quota limits (instead of using Maui+Gold). Can't<br class=""> remember anymore if I was surprised or just sad. :D<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> We use a 14 days HalfLife in slurm.conf:<br class=""> PriorityDecayHalfLife=14-0<br class=""><br class=""> Since our longest running jobs can run only 7 days, maybe our limits<br class=""> never get reduced as you describe?</blockquote><br class=""> <br class=""> The accumulated usage is reduced every 5 minutes (by default; see<br class=""> PriorityCalcPeriod). The reduction is done by multiplying the<br class=""> accumulated usage by a number slightly less than 1. The number is<br class=""> chosen so that the accumulated usage is reduced to 50 % after<br class=""> PriorityDecayHalfLife (given that you don't run anything more in<br class=""> between, of course). With a halflife of 14 days and the default calc<br class=""> period, that number is very close to 1 (0.9998281 if my calculations are<br class=""> correct :).<br class=""> <br class=""> Note: I read all about these details on the schedmd web pages some years<br class=""> ago. I cannot find them again (the parts about the multiplication with<br class=""> a number smaller than 1 to get the half life), so I might be wrong on<br class=""> some of the details.<br class=""> <br class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""> BTW, I've written a handy script for displaying user limits in a<br class=""> readable format:<br class=""> <a href="https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits" class="" target="_blank">https://github.com/OleHolmNielsen/Slurm_tools/tree/master/showuserlimits</a><br data-mce-bogus="1"></blockquote><br class=""> <br class=""> Nice!<br class=""> <br class=""> --<br class=""> B/H<br class=""><br class=""></blockquote></div></div></div></div></blockquote></div><br class=""></div><br></blockquote></div></div></body></html>