<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; } p.MsoNormal, p.MsoNoSpacing { margin: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:14pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p></p>
<div>Hi Aravindh</div>
<div><br>
</div>
<div>For our small 3 node cluster I've hacked together a per-node python script that collects current and peak cpu, memory and scratch disk usage data on all jobs running on the cluster and builds a fairly simple web-page based on it. It shouldn't be hard to
make it store those data points over time, then shove them through an R script to plot the usage:<br>
</div>
<div><br>
</div>
<div><a href="https://github.com/shawarden/simple-web">https://github.com/shawarden/simple-web</a><br>
</div>
<div><br>
Cheers,<br>
</div>
<div> Sam<br>
</div>
<div><br>
</div>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div class="BodyFragment">
<hr style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:center">
<div class="PlainText">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right"><span style="font-size:large">Sam Hawarden</span></div>
<font size="4">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Assistant Research Fellow</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Pathology Department</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Dunedin School of Medicine</div>
</font></div>
</div>
</div>
</div>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Aravindh Sampathkumar <aravindh@fastmail.com><br>
<b>Sent:</b> Monday, 10 December 2018 02:39<br>
<b>To:</b> slurm-users@lists.schedmd.com<br>
<b>Subject:</b> [slurm-users] CPU & memory usage summary for a job</font>
<div> </div>
</div>
<div>
<div style="font-size:16px">Hi All.<br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">I was wondering if anybody has thought of or hacked around a way to record CPU and memory consumption of a job during its entire duration and give a summary of the usage pattern within that job? <br>
</div>
<div style="font-size:16px">Not the MaxRSS and CPU Time that already gets reported for every job. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">I'm thinking more like a chart of CPU utilisation, memory usage, and disk usage on a per second basis or something like that. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">Asking because some of my users have no clue about the resource consumption of their jobs, and just blindly ask for way more resources as "safe" option. It would be a nice way for users to know simple things like - they asked for
8 cores, but their job ran on just 1 core the entire time because a library they used is single core limited. <br>
</div>
<div style="font-size:16px">We use Cgroups for process accounting and limiting job's cpu and memory usage. We also use QoS for limiting resource reservations at user level. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div id="sig56753105">
<div class="signature">--<br>
</div>
<div class="signature"> Aravindh Sampathkumar<br>
</div>
<div class="signature"> aravindh@fastmail.com<br>
</div>
<div class="signature"><br>
</div>
</div>
<div style="font-size:16px"><br>
</div>
</div>
</div>
</body>
</html>