<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.0pt; line-height:1.3; color:#1F497D">
<div>I've been tracking down people at SC17 and talking about graphing user jobs with them. There's a definite consensus that I should be using influxdb to store the data. After SC17 I'm going to rebuild my setup and write a better how-to. <br>
<br>
The advantage of my current setup is the only requirement is to be running Slurm with cgroups. <br>
<br>
The better and more scalable solution is to have it written in C and managed by the slurmd process on the nodes themselves.<br>
<br>
I think I may provide a Dockerfile later that will spin everything up automatically. Then the only requirement is a crontab entry to run a shell script on your nodes to push data to your Docker instance. <br>
<br>
Carlos, I'd definitely like to take a look at your setup, especially if you can segregate users so they cannot see another users job metrics.<br>
<br>
Nick McCollum<br>
</div>
<div><br>
</div>
<div id="signature-x" class="signature_editor" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.0pt; color:#1F497D">
Sent from <a href="http://www.9folders.com/" style="text-decoration:none; color:#009BDF">
Nine</a><br>
</div>
</div>
<hr style="border:none; height:1px; color:#E1E1E1; background-color:#E1E1E1">
<div style="border:none; padding:3.0pt 0cm 0cm 0cm"><span style="font-size:11.0pt; font-family:Calibri,Arial,Helvetica,sans-serif"><b>From:</b> Carlos Fenoy <minibit@gmail.com><br>
<b>Sent:</b> Nov 15, 2017 10:15 AM<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] Graphing job metrics<br>
</span></div>
<br type="attribution">
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>I developed a plugin around 1.5 years ago that uses the profiling feature of slurm to collect the resource usage information and sends it to influxdb. This is not yet merged in the official slurm release, but it may be in the next 18.x release. If you
want to test this there is a branch in the schedm github repo (<a href="https://github.com/SchedMD/slurm/tree/influxdb">https://github.com/SchedMD/slurm/tree/influxdb</a>)</div>
<div><br>
</div>
<div>We've had this running since I created it in some mid-sized clusters with 10's of thousands of jobs per day without an issue. We have a retention policy of 7 days in influx to avoid collecting too much data. We provide then a grafana dashboard for the
users where they can filter by jobid to see the CPU usage and Memory usage of their jobs.</div>
<div><br>
</div>
<div>If you need more details, I'll be glad to answer your questions.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Carlos</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 14, 2017 at 6:10 PM, Nicholas McCollum <span dir="ltr">
<<a href="mailto:nmccollum@asc.edu" target="_blank">nmccollum@asc.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
All,<br>
<br>
I went to the SchedMD booth last night and talked with the guys. Tim told me<br>
that the Barcelona Supercomputing Center is working on something similar. I am<br>
going to try to meet with their Slurm person and compare notes.<br>
<br>
I'm also going to look into trying InfluxDB instead of Graphite at the<br>
recommendation of some people for performance improvements when querying<br>
hundreds of jobs at the same time.<br>
<br>
If anyone wants a specific time to meet, just e-mail me directly. I will be at<br>
the SC17 convention center all week.<br>
<span class="im HOEnZb"><br>
---<br>
<br>
Nicholas McCollum<br>
HPC Systems Administrator<br>
Alabama Supercomputer Authority<br>
<br>
</span>
<div class="HOEnZb">
<div class="h5">On Tue, Nov 14, 2017 at 11:12:46AM +0000, Simon Flood wrote:<br>
> On 14/11/17 10:58, Chris Samuel wrote:<br>
><br>
> > Yup, certainly interest here!<br>
><br>
> Ditto.<br>
> --<br>
> Simon Flood<br>
> HPC System Administrator<br>
> University of Cambridge Information Services<br>
> United Kingdom<br>
><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">--<br>
Carles Fenoy<br>
</div>
</div>
</div>
</body>
</html>