<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto">You can also use the influxdb profiling plugin I developed that’s included in the latest slurm version. It will provide live cpu and memory usage per task, step, host and job. You can then provide a grafana dashboard to display the live metrics<div><br></div><div>Regards,</div><div>Carlos<br><br><div id="AppleMailSignature">Sent from my iPhone</div><div><br>On 9 Dec 2018, at 14:39, Aravindh Sampathkumar <<a href="mailto:aravindh@fastmail.com">aravindh@fastmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<title></title>
<div style="font-size:16px;">Hi All.<br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">I was wondering if anybody has thought of or hacked around a way to record CPU and memory consumption of a job during its entire duration and give a summary of the usage pattern within that job? <br></div>
<div style="font-size:16px;">Not the MaxRSS and CPU Time that already gets reported for every job. <br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">I'm thinking more like a chart of CPU utilisation, memory usage, and disk usage on a per second basis or something like that. <br></div>
<div style="font-size:16px;"><br></div>
<div style="font-size:16px;">Asking because some of my users have no clue about the resource consumption of their jobs, and just blindly ask for way more resources as "safe" option. It would be a nice way for users to know simple things like - they asked for 8 cores, but their job ran on just 1 core the entire time because a library they used is single core limited. <br></div>
<div style="font-size:16px;">We use Cgroups for process accounting and limiting job's cpu and memory usage. We also use QoS for limiting resource reservations at user level. <br></div>
<div style="font-size:16px;"><br></div>
<div id="sig56753105"><div class="signature">--<br></div>
<div class="signature"> Aravindh Sampathkumar<br></div>
<div class="signature"> <a href="mailto:aravindh@fastmail.com">aravindh@fastmail.com</a><br></div>
<div class="signature"><br></div>
</div>
<div style="font-size:16px;"><br></div>
</div></blockquote></div></body></html>