<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you! That finally tells me where the real billing minutes (and also the used cpu minutes that I was using to bill in my older version of slurm) are stored. I’m not sure how I was to even know GrpTRESRaw is supposed to exist... I see
no mention of it unless I specifically query it or in the docs. That’s a great start.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"># sshare -A test -l -o GrpTRESRaw%70<o:p></o:p></p>
<p class="MsoNormal"> GrpTRESRaw
<o:p></o:p></p>
<p class="MsoNormal"> -----------------------------------------------------
<o:p></o:p></p>
<p class="MsoNormal"> cpu=360,mem=0,energy=0,node=360,billing=60<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Now I have something to query to present to the users in the future.<o:p></o:p></p>
<p class="MsoNormal">So now the issue remains on why I can’t use decimals to bill for time…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">Thanks!<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">-John</span><o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "Thomas M. Payerle" <payerle@umd.edu><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Friday, April 27, 2018 at 11:39 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] New Billing TRES Issue<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><a name="_MailOriginalBody">I have not had a chance to play with the newest Slurm, but I would suggest looking at GrpTRESRaw, which is supposed to gather the usage by TRES (in TRES-minutes).<o:p></o:p></a></p>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">So if there is a billing TRES in GrpTRESRaw, that might be what you want.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On Fri, Apr 27, 2018 at 11:21 AM, Roberts, John E. <</span><a href="mailto:jeroberts@anl.gov" target="_blank"><span style="mso-bookmark:_MailOriginalBody">jeroberts@anl.gov</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="mso-bookmark:_MailOriginalBody">Hi,<br>
<br>
I'm testing the newest version of Slurm and I'm seeing an issue when using the newer billing TRES to charge for cpu time on a partition. I've seen that billing should be used now instead of cpu in order to properly use the "TRESBillingWeights" option on a partition.
<br>
<br>
In my test case, I gave an account 2 hours of billing time. I used 1 hour of this while setting the partition to TRESBillingWeights="CPU=1.0". It seemed to have billed properly.<br>
Next, I set on the same partition TRESBillingWeights="CPU=0.5". I ran several jobs, but the billing never seemed to increase. RawUsage, however, did increment correctly.<br>
<br>
Here's an examples of sshare reporting no billing run minutes, when CPU=0.5 and I start a job with a walltime of 1 hour. Even though the RawUsage is well past 2 hours, a job can still run when it shouldn't.<br>
<br>
# sshare -A test -l -o RawUsage,GrpTRESMins,TRESRunMins%60<br>
RawUsage GrpTRESMins TRESRunMins
<br>
----------- ------------------------------ -----------------------------------------------------
<br>
11068 billing=120 cpu=60,mem=0,energy=0,node=60,billing=0<br>
<br>
If I set CPU=1.0 and start say a job for 2 hours, I get this in the logs:<br>
debug2: Job 32 being held, the job is at or exceeds assoc 239(test/(null)/(null)) group max tres(billing) minutes of 120 of which 60 are still available but request is for 120 (plus 0 already in use) tres minutes (request tres count 1)<br>
<br>
This makes sense because I previously ran a job at the weight of 1.0 for an hour so it "billed" for 1 hour at that time. How can I query the "available" billing hours if it's not RawUsage?<br>
<br>
Going back to setting billing CPU weight to 0.5, the logs seem to be inconsistent too. In this first line, it shows the right thing:<br>
debug: TRES Weight: cpu = 1.000000 * 0.500000 = 0.500000<br>
<br>
but not a few lines down:<br>
debug2: acct_policy_job_begin: after adding job 45, assoc 239(test/(null)/(null)) grp_used_tres_run_secs(billing) is 0<br>
<br>
Again, RawUsage increases correctly, but Slurm is using some other field for billing to determine if a job can run.<br>
<br>
My questions are: How can I set CPU billing to be less than 1 and how can I make sure jobs don't run if they are out of time in this case? What is Slurm using for billing, because it's clearly not RawUsage? Am I simply misunderstanding the billing and/or weights
fields?<br>
<br>
Thanks for any help...<o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><br>
<br clear="all">
<br>
-- <o:p></o:p></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">Tom Payerle <br>
DIT-ACIGS/Mid-Atlantic Crossroads </span><a href="mailto:payerle@umd.edu" target="_blank"><span style="mso-bookmark:_MailOriginalBody">payerle@umd.edu</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">5825 University Research Park (301) 405-6135<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">University of Maryland<br>
College Park, MD 20740-3831<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>