<div dir="ltr"><div><div><div><div>Hi Gareth,<br></div>Your assessment is also what I would have thought MaxRSS should be the maximum of the sum of all RSS in a sample, swap and shared memory does complicate things but I think most people expect jobs to only be killed if their RSS exceeds their memory request.<br><br></div>That being said as far as I understand the current slurm reporting mechanisms there is actually no way to get the total MaxRSS of a job but only of whatever step/subjob/thread was largest in memory.<br></div>Thanks,<br></div>Eli<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 17, 2018 at 4:03 PM, <span dir="ltr"><<a href="mailto:Gareth.Williams@csiro.au" target="_blank">Gareth.Williams@csiro.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-AU">
<div class="m_7559271900767106106WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I think the situation is likely to be a little different. Let’s consider a fortran program that statically or dynamically defines large
arrays. This defines a virtual memory size – like declaring that this is the maximum amount of memory you might use if you fill the arrays. That amount of real memory + swap must be available for the program to run – after all, you might use that amount… Speaking
loosely, linux has a soft memory allocation policy so memory may not actually be allocated until it is used. If the program happens to read a smaller dataset and the arrays are not filled then the resident set size may be significantly smaller than the virtual
memory size. Further, memory swapped doesn’t count to the RSS so it might be even smaller. Effectively RSS for a process is the actual footprint in RAM. It will change over the life of the process/job and slurm will track the maximum (MaxRSS). I’d actually
expect MaxRSS to be the maximum of the sum of RSS of known processes as sampled periodically through the job – but I’m guessing. This should apply reasonably to parallel jobs if the sum spans nodes (or it wouldn’t be the first batch system to only effectively
account for the first allocated node). The whole linux memory tracking/accounting system has gotchas as shared memory (say for library code) has to be accounted for somewhere, but we can reasonably assume in HPC that memory use is dominated by unique computational
working set data – so MaxRSS is a good estimate of how much RAM is needed to run a given job.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Gareth<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif" lang="EN-US"> slurm-users [mailto:<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@<wbr>lists.schedmd.com</a>]
<b>On Behalf Of </b>E.S. Rosenberg<br>
<b>Sent:</b> Tuesday, 17 April 2018 10:42 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a><wbr>><br>
<b>Subject:</b> Re: [slurm-users] Way MaxRSS should be interpreted<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi Loris,<u></u><u></u></p>
</div>
<p class="MsoNormal">Thanks for your explanation!<u></u><u></u></p>
</div>
<p class="MsoNormal">I would have interpreted as max(sum()).<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Is there a way to get max(sum()) or at least sum form of sum()? The assumption that all processes are peaking at the same value is not a valid one unless all threads have essentially the same workload...<u></u><u></u></p>
</div>
<p class="MsoNormal">Thanks again!<u></u><u></u></p>
</div>
<p class="MsoNormal">Eli<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Apr 17, 2018 at 2:09 PM, Loris Bennett <<a href="mailto:loris.bennett@fu-berlin.de" target="_blank">loris.bennett@fu-berlin.de</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Eli,<br>
<br>
"E.S. Rosenberg" <<a href="mailto:esr%2Bslurm-dev@mail.hebrew.edu" target="_blank">esr+slurm-dev@mail.hebrew.edu</a><wbr>> writes:<br>
<br>
> Hi fellow slurm users,<br>
> We have been struggling for a while with understanding how MaxRSS is reported.<br>
><br>
> This because jobs often die with MaxRSS not even approaching 10% of the requested memory sometimes.<br>
><br>
> I just found the following document:<br>
> <a href="https://research.csc.fi/-/a" target="_blank">https://research.csc.fi/-/a</a><br>
><br>
> It says:<br>
> "maxrss = maximum amount of memory used at any time by any process in that job. This applies directly for serial jobs. For parallel jobs you need to multiply with the number of cores (max 16 or 24 as this is<br>
> reported only for that node that used the most memory)"<br>
><br>
> While 'man sacct' says:<br>
> "Maximum resident set size of all tasks in job."<br>
><br>
> Which explanation is correct? How should I be interpreting MaxRSS?<br>
<br>
As far as I can tell, both explanations are correct, but the<br>
text in 'man acct' is confusing.<br>
<br>
"Maximum resident set size of all tasks in job."<br>
<br>
is analogous to<br>
<br>
"maximum height of all people in the room"<br>
<br>
rather than <br>
<br>
"total height of all people in the room"<br>
<br>
More specifically it means<br>
<br>
"Maximum individual resident set size out of the group of resident set<br>
sizes associated with all tasks in job."<br>
<br>
It doesn't mean<br>
<br>
"Sum of the resident set sizes of all the tasks"<br>
<br>
I'm a native English-speaker and I keep on stumbling over this in 'man<br>
sacct' and then remembering that I have already worked out how it was<br>
supposed to be interpreted.<br>
<br>
My suggestion for improving this would be<br>
<br>
"Maximum individual resident set size of all resident set sizes<br>
associated with the tasks in job."<br>
<br>
It's a little clunky, but I hope it is clearer.<br>
<br>
Cheers,<br>
<br>
Loris<br>
<span style="color:#888888"><br>
<span class="m_7559271900767106106hoenzb">-- </span><br>
<span class="m_7559271900767106106hoenzb">Dr. Loris Bennett (Mr.)</span><br>
<span class="m_7559271900767106106hoenzb">ZEDAT, Freie Universität Berlin Email <a href="mailto:loris.bennett@fu-berlin.de" target="_blank">
loris.bennett@fu-berlin.de</a></span></span><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>