<div dir="ltr">Would job profiling with HDF5 work as well? <a href="https://slurm.schedmd.com/hdf5_profile_user_guide.html" target="_blank">https://slurm.schedmd.com/hdf5_profile_user_guide.html</a><div><br></div><div>Jacob <br><div class="gmail-yj6qo gmail-ajU" style="outline:none;padding:10px 0px;width:22px;margin:2px 0px 0px"><br class="gmail-Apple-interchange-newline"></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Sun, Dec 9, 2018 at 4:17 PM Sam Hawarden <<a href="mailto:sam.hawarden@otago.ac.nz">sam.hawarden@otago.ac.nz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr" style="font-size:14pt;color:rgb(0,0,0);background-color:rgb(255,255,255);font-family:Calibri,Arial,Helvetica,sans-serif">
<p></p>
<div>Hi Aravindh</div>
<div><br>
</div>
<div>For our small 3 node cluster I've hacked together a per-node python script that collects current and peak cpu, memory and scratch disk usage data on all jobs running on the cluster and builds a fairly simple web-page based on it. It shouldn't be hard to
make it store those data points over time, then shove them through an R script to plot the usage:<br>
</div>
<div><br>
</div>
<div><a href="https://github.com/shawarden/simple-web" target="_blank">https://github.com/shawarden/simple-web</a><br>
</div>
<div><br>
Cheers,<br>
</div>
<div> Sam<br>
</div>
<div><br>
</div>
<div id="gmail-m_2289635656461447970Signature">
<div name="divtagdefaultwrapper">
<div class="gmail-m_2289635656461447970BodyFragment">
<hr style="font-family:Calibri,Arial,Helvetica,sans-serif;text-align:center">
<div class="gmail-m_2289635656461447970PlainText">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;text-align:right"><span style="font-size:large">Sam Hawarden</span></div>
<font size="4">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;text-align:right">Assistant Research Fellow</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;text-align:right">Pathology Department</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;text-align:right">Dunedin School of Medicine</div>
</font></div>
</div>
</div>
</div>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block;width:98%">
<div id="gmail-m_2289635656461447970divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Aravindh Sampathkumar <<a href="mailto:aravindh@fastmail.com" target="_blank">aravindh@fastmail.com</a>><br>
<b>Sent:</b> Monday, 10 December 2018 02:39<br>
<b>To:</b> <a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><br>
<b>Subject:</b> [slurm-users] CPU & memory usage summary for a job</font>
<div> </div>
</div>
<div>
<div style="font-size:16px">Hi All.<br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">I was wondering if anybody has thought of or hacked around a way to record CPU and memory consumption of a job during its entire duration and give a summary of the usage pattern within that job? <br>
</div>
<div style="font-size:16px">Not the MaxRSS and CPU Time that already gets reported for every job. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">I'm thinking more like a chart of CPU utilisation, memory usage, and disk usage on a per second basis or something like that. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div style="font-size:16px">Asking because some of my users have no clue about the resource consumption of their jobs, and just blindly ask for way more resources as "safe" option. It would be a nice way for users to know simple things like - they asked for
8 cores, but their job ran on just 1 core the entire time because a library they used is single core limited. <br>
</div>
<div style="font-size:16px">We use Cgroups for process accounting and limiting job's cpu and memory usage. We also use QoS for limiting resource reservations at user level. <br>
</div>
<div style="font-size:16px"><br>
</div>
<div id="gmail-m_2289635656461447970sig56753105">
<div class="gmail-m_2289635656461447970signature">--<br>
</div>
<div class="gmail-m_2289635656461447970signature"> Aravindh Sampathkumar<br>
</div>
<div class="gmail-m_2289635656461447970signature"> <a href="mailto:aravindh@fastmail.com" target="_blank">aravindh@fastmail.com</a><br>
</div>
<div class="gmail-m_2289635656461447970signature"><br>
</div>
</div>
<div style="font-size:16px"><br>
</div>
</div>
</div>
</div>
</blockquote></div>