<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Dear Jürgen,</p>
<p><br>
</p>
<p>man thanks for your reply, and your thoughts! <br>
</p>
<p><br>
</p>
<p>What you say makes deductively much sense to me ;) It is only confusing in respect to the SchedMD documentation, as I pointed out. (MaxRSSTask is then distinguished to be what?)<br>
</p>
<p>So, if you are right, this includes really everything, including the OS. And all dynamic libraries double-counted? Hm. Really not a good metric then.</p>
<p>I then wonder why Slurm does not use immediately the /proc/meminfo information. Seems that this one should be much more precise under these circumstances ... (at least I hope the Linux kernel people do a much better job here ;) )<br>
</p>
<p><br>
</p>
<p>About PSS, I didn't know ... until now. Maybe it's worth some tests ... if I can convince our admins to collaborate ;)</p>
<p><br>
</p>
<p><a href="https://bugs.schedmd.com/show_bug.cgi?id=9010" class="OWAAutoLink">https://bugs.schedmd.com/show_bug.cgi?id=9010</a> is very interesting. Seems as it exactly resembles my observation. And therein, cgroup seems to be considered as a cause. Although
we also have cgroups active, afaik, I wonder a bit how this might influence counting memory occupation.</p>
<p><br>
</p>
<p>It is maybe not a very good idea generally. But for the moment, I use "\time -v" to scrutinize rank-wise MaxRSS. I'm unsure about its thread-safety. But the values I usually observe seem reasonable. And here, there is no doubt that it is indeed MPI rank-wise
(of the code, I really want to investigate ... I hope so ...). But I never summed up all these values for a node. Maybe it's a good occasion to try now. If that's also larger than 100% ... ;)</p>
<p><br>
</p>
<p>Kind regards,</p>
<p>Martin<br>
</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Von:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> im Auftrag von Juergen Salk <juergen.salk@uni-ulm.de><br>
<b>Gesendet:</b> Mittwoch, 2. November 2022 14:25<br>
<b>An:</b> Slurm User Community List<br>
<b>Betreff:</b> Re: [slurm-users] slurm accounting shows more MaxRSS than physically available memory</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Hi Martin,<br>
<br>
to my very best knowledge MaxRSS does report aggregated memory consumption <br>
of all tasks but including all the shared libraries that the individual <br>
processes uses, even though a shared library is only loaded into memory <br>
once regardless of how many processes use it.<br>
<br>
So shared libraries do count multiple times (once for every individual <br>
process) to MaxRSS when summed up. This can even result in a higher <br>
MaxRSS value reported by sacct than the total amount of memory that is <br>
physically available.<br>
<br>
For exactly that reason we have decided to use<br>
`JobAcctGatherParams=UsePss´ in slurm.conf as we think proportional<br>
set size (PSS) is more useful than RSS because when the PSS for all<br>
processes are summed together, that seems to be a better<br>
representation for the "true" total memory consumption of the job.<br>
<br>
Also see:<br>
<br>
<a href="https://en.wikipedia.org/wiki/Proportional_set_size">https://en.wikipedia.org/wiki/Proportional_set_size</a><br>
<a href="https://bugs.schedmd.com/show_bug.cgi?id=9010">https://bugs.schedmd.com/show_bug.cgi?id=9010</a><br>
<br>
I would also be interested what others think.<br>
<br>
Best regards<br>
Jürgen<br>
<br>
<br>
<br>
* Ohlerich, Martin <Martin.Ohlerich@lrz.de> [221102 11:53]:<br>
> Dear "Commiserates".<br>
> <br>
> I wonder a bit about the meaning of MaxRSS. The documentation says:<br>
> "Maximum resident set size of all tasks in job."<br>
><br>
> To what refers here "maximum"? The maximum over job period, I<br>
> understand hopefully correctly. But it does not seem to be the size<br>
> of all tasks (summed up, so-to-speak), but the maximum size of RSS<br>
> of that task with the largest RSS of all tasks during the job's<br>
> period. Right?<br>
> <br>
> In any case, I observed something like this:<br>
> <br>
> login08:~> sacct -j 2408392 -o 'maxrss,maxrssnode%20'<br>
> MaxRSS MaxRSSNode<br>
> ---------- --------------------<br>
> 102124920K i02r09c03s02<br>
> <br>
> ... so 102 GB if I counted the decimal positions correctly. On the other hand, for specifically this node, I actually only have<br>
> <br>
> i02r09c03s02:~> cat /proc/meminfo<br>
> MemTotal: 98436736 kB<br>
> <br>
> i.e. 98 GB RAM ...<br>
> <br>
> Does anybody know whether there is a reasonable explanation how this can be? Specifically is the situation even worse, if MaxRSS is the maximum RSS of only one task (rank) on that node. What about the other tasks (which certainly also consume memory). And
also the OS is quite large on these disk-less compute nodes.<br>
> <br>
> Would be nice if you could share any ideas about my finding. Thank you!<br>
> Kind regards,<br>
> Martin<br>
> <br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>