<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I would recommend standing up an instance of XDMod as it handles
most of this for you in its summary reports.</p>
<p><br>
</p>
<p><a class="moz-txt-link-freetext" href="https://open.xdmod.org/10.0/index.html">https://open.xdmod.org/10.0/index.html</a></p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 5/3/23 2:05 PM, Joseph Francisco
Guzman wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DM6PR18MB3635B00CAF3BBD59EADEDC70BD6C9@DM6PR18MB3635.namprd18.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<style type="text/css" style="display:none;">P {margin-top:0;margin-bottom:0;}</style>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof
ContentPasted0">
Good morning,<br>
<br>
We have at least one billed account right now, where the
associated researchers are able to submit jobs that run against
our normal queue with fairshare, but not for an academic
research purpose. So we'd like to accurately calculate their CPU
hours. We are currently using a script to query the db with
sacct and sum up the value of ElapsedRaw * AllocCPUS for all
jobs. But this seems limited, because requeueing will create
what the sacct man page calls duplicates. By default jobs
normally get requeued only if there's something outside of the
user's control like a NODE_FAIL or an scontrol command to
requeue it manually, though I think users can requeue things
themselves, it's not a feature we've seen our researchers use.<br>
<br>
However with the new scrontab feature, whenever the cron is
executed more than once, sacct reports that the previous jobs
are "requeued" and are only visible by looking up duplicates. I
haven't seen any billed account use requeueing or scrontab yet,
but it's clear to me that it could be significant once
researchers start using scrontab more. Scrontab has existed
since one of the releases from 2020 I believe, but we enabled it
this year and see it as much more powerful than the traditional
linux crontab.<br>
<br>
What would be the best way to more thoroughly calculate
ElapsedRaw * AllocCPUS, to account for duplicates, but
optionally ignore unintentional requeueing like from a
NODE_FAIL?<br>
<br>
Here's the main loop of the simple bash script I have now:<br>
<br>
while IFS='|' read -r end elapsed cpus; do
<div class="ContentPasted0"> # if a job crosses the month
barrier</div>
<div class="ContentPasted0"> # the entire bill will be put
under the 2nd month</div>
<div class="ContentPasted0"> year_month="${end:0:7}"</div>
<div class="ContentPasted0"> if [[ ! "$elapsed" =~ ^[0-9]+$
]] || [[ ! "$cpus" =~ ^[0-9]+$ ]]; then</div>
<div class="ContentPasted0"> continue</div>
<div class="ContentPasted0"> fi</div>
<div class="ContentPasted0"> core_seconds["$year_month"]=$((
core_seconds["$year_month"] + (elapsed * cpus) ))</div>
<div class="ContentPasted0 elementToProof">done < <(sacct
-a -A "$SLURM_ACCOUNT" \</div>
<div class="ContentPasted0"> -S "$START_DATE" \</div>
<div class="ContentPasted0"> -E "$END_DATE" \</div>
-o End,ElapsedRaw,AllocCPUS -X -P --noheader)<br>
<br>
Our slurmdbd is configured to keep 6 months of data.<br>
<br>
It make senses to loop through the jobids instead, using sacct's
-D/--duplicates option each time to reveal the hidden duplicates
in the REQUEUED state, but I'm interested if there are
alternatives or if I'm missing anything here.<br>
<br>
Thanks,<br>
<br>
Joseph</div>
<div class="elementToProof">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:
12pt; font-family: Calibri, Helvetica, sans-serif; color:
rgb(0, 0, 0);">
<p><span>--------------------------------------------------------------</span><br
style="font-family: wf_segoe-ui_normal, "Segoe
UI", "Segoe WP", Tahoma, Arial,
sans-serif, serif, EmojiFont; font-size: 13.3333px;
color: rgb(33, 33, 33);">
<font style="color: rgb(33, 33, 33);"
face="wf_segoe-ui_normal, Segoe UI, Segoe WP, Tahoma,
Arial, sans-serif, serif, EmojiFont"><span
style="font-size:13.3333px">Joseph F. Guzman - ITS
(Advanced Research Computing)</span></font></p>
<p><font style="color: rgb(33, 33, 33);"
face="wf_segoe-ui_normal, Segoe UI, Segoe WP, Tahoma,
Arial, sans-serif, serif, EmojiFont"><span
style="font-size:13.3333px">Northern Arizona
University</span></font></p>
<p><font style="color: rgb(33, 33, 33);"
face="wf_segoe-ui_normal, Segoe UI, Segoe WP, Tahoma,
Arial, sans-serif, serif, EmojiFont"><span
style="font-size:13.3333px"><a class="moz-txt-link-abbreviated" href="mailto:Joseph.F.Guzman@nau.edu">Joseph.F.Guzman@nau.edu</a></span></font></p>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>