<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Before you get all excited about it, we have had a terrible time trying to get gppu metrics. Finally abandoned and switch to Grafana, Prometheus influx. Good luck to you though.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of "Heckes, Frank" <heckes@mps.mpg.de><br>
<b>Reply-To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Date: </b>Wednesday, April 14, 2021 at 1:56 AM<br>
<b>To: </b>Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] derived counters<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><span lang="EN-GB">Hi all, many thanks for all hints. The link in the latest pointing points to an impressive switch-board.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB">-Frank</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Renfro, Michael<br>
<b>Sent:</b> Tuesday, 13 April 2021 19:25<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] derived counters<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I'll never miss an opportunity to plug XDMoD for anyone who doesn't want to write custom analytics for every metric. I've managed to get a little bit into its API to extract current values for number of jobs completed and the number of
CPU-hours provided, and insert those into a single slide presentation for introductory meetings.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">You can see a working version of it for the NSF XSEDE facilities at
<a href="https://xdmod.ccr.buffalo.edu">https://xdmod.ccr.buffalo.edu</a><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <</span><a href="mailto:slurm-users-bounces@lists.schedmd.com"><span style="font-size:12.0pt">slurm-users-bounces@lists.schedmd.com</span></a><span style="font-size:12.0pt;color:black">> on behalf
of Hadrian Djohari <</span><a href="mailto:hxd58@case.edu"><span style="font-size:12.0pt">hxd58@case.edu</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Tuesday, April 13, 2021 at 8:11 AM<br>
<b>To: </b>Slurm User Community List <</span><a href="mailto:slurm-users@lists.schedmd.com"><span style="font-size:12.0pt">slurm-users@lists.schedmd.com</span></a><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Re: [slurm-users] derived counters</span><o:p></o:p></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<div>
<p class="MsoNormal">Hi Frank, <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">A way to get "how long jobs wait in the queue" is to import the data to XDMOD (<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fopen.xdmod.org%2F9.0%2Findex.html&data=04%7C01%7Crenfro%40tntech.edu%7C38d51462bef94bee8a9708d8fe7db3d9%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637539163146606550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5d82B%2BR1JhiuuUn0is%2FWojmMlt87YpzLnBI%2FOtpokTY%3D&reserved=0">https://open.xdmod.org/9.0/index.html</a>).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The nifty reporting tool has many features to make it easier for us to report out the cluster usage.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Hadrian<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Apr 13, 2021 at 8:08 AM Heckes, Frank <<a href="mailto:heckes@mps.mpg.de">heckes@mps.mpg.de</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-bottom:12.0pt">Hello Ole,<br>
<br>
> >> -----Original Message-----<br>
> >>> * (average) queue length for a certain partition<br>
> <br>
> I wonder what exactly does your question mean? Maybe the number of jobs or<br>
> CPUs in the Pending state? Maybe relative to the number of CPUs in the<br>
> partition?<br>
> <br>
This result from a mgmt. - question. How long jobs have to wait (in s, min, h, day) before they getting executed and
<br>
how many jobs are waiting (are queued) for each partition in a certain time interval.
<br>
The first one is easy to find with sacct and submit, start counts + difference + averaging.<br>
The second is a bit cumbersome, so I wonder whether a 'solution' is already around. The easiest way is to monitor from the beginning and store the squeue ouput for later evaluation. Unfortunately I didn’t do that.<br>
<br>
Cheers,<br>
-Frank<br>
<br>
> The "slurmacct" command prints (possibly for a specified partition) the<br>
> average job waiting time while Pending in the queue, but not the queue length<br>
> information.<br>
> <br>
> It may be difficult to answer your question from the Slurm database. The sacct<br>
> command displays accounting data for all jobs and job steps, but not directly<br>
> for partitions.<br>
> <br>
> There are other Slurm monitoring tools which perhaps can supply the data you<br>
> are looking for. You could ask this list again.<br>
> <br>
> /Ole<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hadrian Djohari<br>
Manager of Research Computing Services, [U]Tech<br>
Case Western Reserve University<br>
(W): 216-368-0395<br>
(M): 216-798-7490 <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>