<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Here's seff output, if it makes any difference. In any case, the exact same job was run by the user on their laptop with 16 GB RAM with no problem.</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Job ID: 83387
<div>Cluster: picotte</div>
<div>User/Group: foob/foob</div>
<div>State: OUT_OF_MEMORY (exit code 0)</div>
<div>Nodes: 1</div>
<div>Cores per node: 16</div>
<div>CPU Utilized: 06:50:30</div>
<div>CPU Efficiency: 11.96% of 2-09:10:56 core-walltime</div>
<div>Job Wall-clock time: 03:34:26</div>
<div>Memory Utilized: 1.54 GB</div>
Memory Efficiency: 1.21% of 128.00 GB<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:'Courier New',monospace">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText"></div>
<div class="PlainText" style="font-family:"Courier New",monospace; font-size:13.3333px">
</div>
<span id="ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div>dwc62@drexel.edu 215.571.4335 (o)</div>
<div>For URCF support: urcf-support@drexel.edu</div>
<div>https://proteusmaster.urcf.drexel.edu/urcfwiki</div>
<div>github:prehensilecode</div>
</div>
<span id="ms-rterangepaste-end"></span>
<div class="PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Paul Edmon <pedmon@cfa.harvard.edu><br>
<b>Sent:</b> Monday, March 15, 2021 14:02<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Job ended with OUT_OF_MEMORY even though MaxRSS and MaxVMSize are under the ReqMem value</font>
<div> </div>
</div>
<div>
<table width="100%">
<tbody>
<tr>
<td style="border-left:4px solid goldenrod; background:cornsilk; padding:0 3pt">
<p style="font:small-caps bold 100% sans-serif">External.</p>
</td>
</tr>
</tbody>
</table>
<div>
<p>One should keep in mind that sacct results for memory usage are not accurate for Out Of Memory (OoM) jobs. This is due to the fact that the job is typically terminated prior to next sacct polling period, and also terminated prior to it reaching full memory
allocation. Thus I wouldn't trust any of the results with regards to memory usage if the job is terminated by OoM. sacct just can't pick up a sudden memory spike like that and even if it did it would not correctly record the peak memory because the job
was terminated prior to that point.</p>
<p><br>
</p>
<p>-Paul Edmon-</p>
<p><br>
</p>
<div class="x_moz-cite-prefix">On 3/15/2021 1:52 PM, Chin,David wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
Hi, all:</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
I'm trying to understand why a job exited with an error condition. I think it was actually terminated by Slurm: job was a Matlab script, and its output was incomplete. </div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
Here's sacct output:</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
JobID JobName User Partition NodeList Elapsed State ExitCode ReqMem MaxRSS MaxVMSize AllocTRES AllocGRE
<div>-------------------- ---------- --------- ---------- --------------- ---------- ---------- -------- ---------- ---------- ---------- -------------------------------- --------</div>
<div> 83387 ProdEmisI+ foob def node001 03:34:26 OUT_OF_ME+ 0:125 128Gn billing=16,cpu=16,node=1</div>
<div> 83387.batch batch node001 03:34:26 OUT_OF_ME+ 0:125 128Gn 1617705K 7880672K cpu=16,mem=0,node=1</div>
83387.extern extern node001 03:34:26 COMPLETED 0:0 128Gn 460K 153196K billing=16,cpu=16,node=1<br>
</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
Thanks in advance,</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
Dave</div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_Signature">
<div>
<div id="x_divtagdefaultwrapper" dir="ltr" style="">
<div class="x_BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="x_PlainText" style=""></div>
<span id="x_ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div><a class="x_moz-txt-link-abbreviated" href="mailto:dwc62@drexel.edu">dwc62@drexel.edu</a> 215.571.4335 (o)</div>
<div>For URCF support: <a class="x_moz-txt-link-abbreviated" href="mailto:urcf-support@drexel.edu">
urcf-support@drexel.edu</a></div>
<div><a class="x_moz-txt-link-freetext" href="https://proteusmaster.urcf.drexel.edu/urcfwiki">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="x_ms-rterangepaste-end"></span>
<div class="x_PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<br>
<p align="Left" style="font-family:Calibri; font-size:10pt; color:#000000; margin:5pt">
Drexel Internal Data<br>
</p>
</blockquote>
</div>
</div>
<br>
<p style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;" align="Left">
Drexel Internal Data<br>
</p>
</body>
</html>