<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 3/12/21 6:37 PM, Sean Crosby wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAFstPEAdRpfCL8RxEQpck=XbhFDjKQkPTkhG5YRBfdj84ozH7w@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
<br>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, 13 Mar 2021 at
08:48, Prentice Bisbal <<a href="mailto:pbisbal@pppl.gov"
moz-do-not-send="true">pbisbal@pppl.gov</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>
<div
style="color:rgb(0,0,0);font-size:12px;text-align:left;font-family:Helvetica,Arial,sans-serif"><b>
<table
style="width:100%;float:left;background-color:lemonchiffon"
cellspacing="0" cellpadding="5" border="1">
<tbody>
<tr>
<td><b>UoM notice: </b>External email. Be
cautious of links, attachments, or
impersonation attempts</td>
</tr>
</tbody>
</table>
</b><br>
</div>
<hr></div>
<p>It sounds like your confusing job steps and tasks. For
an MPI program, tasks and MPI ranks are the same thing.
A slurm job has multiple steps. A single job step could
have only 1 task, while another step in the same job can
use 1,000 tasks. When looking at the amount of memory
for a job, the important number is the largest value of
MaxRSS for all the job steps. Why this important?
Because if you don't request at least this much with
your --mem specification, your job may fail. <br>
</p>
<p>Based on your definition, of aveRSS (I didn't go back
and check the documentation myself), it sounds like
you're doing unnecessary math, since I'm sure Slurm sums
up the individual task max. RSS values for each task to
get MaxRSS, and then divides that by the number of tasks
to get the AveRSS. <br>
</p>
</div>
</blockquote>
<div>This is incorrect. MaxRSS is the maximum amount of RAM
the task that used the most amount of RAM used. That is why
there is then a MaxRSSNode and MaxRSSTask value. MaxRSSNode
is the node the task that used the most amount of RAM was
on, and MaxRSSTask is the task ID of the task that used the
most amount of RAM.</div>
</div>
</div>
</blockquote>
<p>Thanks for the correction. That's what I originally thought, and
then read the definition he provided, which is exactly the same as
in the documentation, and completely misinterpreted it. When I
look at the sacct documentation and see that same definition in
the context of all the all the other MaxRSS values, it's clear I
screwed up. Sorry!</p>
<p>SchedMD should reword that so even out of context it's clear what
it represents. <br>
</p>
<p>When I read "Maximum resident set size of all tasks in job" I
automatically thought "Maximum of the *sum* of the RSSes of each
task.</p>
<p>Prentice<br>
</p>
<blockquote type="cite"
cite="mid:CAFstPEAdRpfCL8RxEQpck=XbhFDjKQkPTkhG5YRBfdj84ozH7w@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>If you are trying to work out the RAM that the job as a
whole used, use TRESUsageInTot</div>
<div><br>
</div>
<div>For a job on our cluster:</div>
<div><br>
</div>
<div># sacct -j 24207294 -o
JobID,Node,AveRSS,MaxRSS,MaxRSSTask,MaxRSSNode,TRESUsageInTot
-p<br>
JobID|NodeList|AveRSS|MaxRSS|MaxRSSTask|MaxRSSNode|TRESUsageInTot|<br>
24207294.0|spartan-bm[055-056,058-059,061-062,085,091-093,096,098-099,104,108,112-117,120-124]|927811665|962245K|3|spartan-bm058|cpu=4784-18:38:23,energy=0,fs/disk=3555263283,mem=217455859K,pages=2438,vmem=434981656K|</div>
<div><br>
</div>
<div>This shows that AveRSS was 884MB, MaxRSS was task 3
running on spartan-bm058, which used 939MB, and all tasks in
total used 212359MB</div>
<div><br>
</div>
<div>Also remember that --mem is a per node memory request. It
is not a per job or a per task memory request.</div>
<div><br>
</div>
<div>Sean<br>
</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p> </p>
<div>On 3/9/21 3:41 AM, <a
href="mailto:xiaojinghu93@163.com" target="_blank"
moz-do-not-send="true">xiaojinghu93@163.com</a> wrote:<br>
</div>
<blockquote type="cite">
<pre>Hi guys,
I would like to calculate the CPU efficiency and Memory efficiency of slurm jobs.
I am having difficulty calculating the real “memory” a job use.
According to slurm, “maxRSS” means "Maximum resident set size of all tasks in job”. If so, how can I get the memory used by a single job? As far as I am concerned, if I need to know the memory used by a single job/jobstep, I need to sum up the memory used for each task. So I think I should use the “aveRSS” field which gives the "average resident set size of all tasks in job”. If I multiply the “aveRSS” with “task”, I should get the real memory a job/jobstep used.
But I studied the code of the “seff” command and it claims to be equivalent to "sacct -P -n -a --format JobID,User,Group,State,Cluster,AllocCPUS,REQMEM,TotalCPU,Elapsed,MaxRSS,ExitCode,NNodes,NTasks -j <job_id>”, which means I should use “maxRSS”.
Can anyone give me some explanation on that?
Very grateful for any help.
Thank you!
Regards,
Xiaojing
</pre>
</blockquote>
</div>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>