<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Dave,<div><br></div><div>Hope you're doing well.</div><div><br></div><div>(...very possible you have already done these things...)<br></div><div><br></div><div>Maybe the logs on the compute node (system and slurmd.log) would yield more info? </div><div><br></div><div>Rolling dice, it may also be worth a look for runaway processes or jobs on that compute node as well as confirm the node is healthy... (No hardware issues, etc.)</div><div><br></div><div>Cheers,</div><div>Chad</div><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">------------------------------------------------------------<u></u><u></u></font></span></p><p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">Chad DeWitt, CISSP | University Research Computing</font></span></p><p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">UNC Charlotte <b>| </b>Office of OneIT</font></span></p><p style="margin:0in 0in 0.0001pt"><font color="#000000" size="2" face="monospace, monospace"><span style="background-color:rgba(255,255,255,0)"><a href="mailto:ccdewitt@uncc.edu" target="_blank">ccdewitt@uncc.edu</a> <b>| </b></span></font><font color="#000000" face="monospace, monospace"><span><a href="https://oneit.uncc.edu" target="_blank">https://oneit.uncc.edu</a></span></font></p><p style="margin:0in 0in 0.0001pt"><span style="background-color:rgba(255,255,255,0)"><font size="2" face="monospace, monospace">------------------------------------------------------------</font></span></p><p style="margin:0in 0in 0.0001pt"><br></p></div></div></div></div></div></div></div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 15, 2021 at 2:50 PM Chin,David <<a href="mailto:dwc62@drexel.edu" target="_blank">dwc62@drexel.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">[<strong><font color="red">Caution</font></strong>: Email from External Sender. Do not click or open links or attachments unless you know this sender.]<br><span> </span>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
One possible datapoint: on the node where the job ran, there were two slurmstepd processes running, both at 100%CPU even after the job had ended.</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:"Courier New",monospace">
<div><font size="2"><span style="font-size:10pt">
<div></div>
<div style="font-family:"Courier New",monospace;font-size:13.3333px">
</div>
<span id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div><a href="mailto:dwc62@drexel.edu" target="_blank">dwc62@drexel.edu</a> 215.571.4335 (o)</div>
<div>For URCF support: <a href="mailto:urcf-support@drexel.edu" target="_blank">urcf-support@drexel.edu</a></div>
<div><a href="https://proteusmaster.urcf.drexel.edu/urcfwiki" target="_blank">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191ms-rterangepaste-end"></span>
<div><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> on behalf of Chin,David <<a href="mailto:dwc62@drexel.edu" target="_blank">dwc62@drexel.edu</a>><br>
<b>Sent:</b> Monday, March 15, 2021 13:52<br>
<b>To:</b> Slurm-Users List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] Job ended with OUT_OF_MEMORY even though MaxRSS and MaxVMSize are under the ReqMem value</font>
<div> </div>
</div>
<div dir="ltr">
<table width="100%">
<tbody>
<tr>
<td style="border-left-width:4px;border-left-style:solid;border-left-color:goldenrod;background-color:cornsilk;padding:0px 3pt">
<p style="font-style:normal;font-variant-caps:small-caps;font-weight:bold;font-stretch:normal;font-size:100%;line-height:normal;font-family:sans-serif">External.</p>
</td>
</tr>
</tbody>
</table>
<div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Hi, all:</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
I'm trying to understand why a job exited with an error condition. I think it was actually terminated by Slurm: job was a Matlab script, and its output was incomplete. </div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Here's sacct output:</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
JobID JobName User Partition NodeList Elapsed State ExitCode ReqMem MaxRSS MaxVMSize AllocTRES AllocGRE
<div>-------------------- ---------- --------- ---------- --------------- ---------- ---------- -------- ---------- ---------- ---------- -------------------------------- --------</div>
<div> 83387 ProdEmisI+ foob def node001 03:34:26 OUT_OF_ME+ 0:125 128Gn billing=16,cpu=16,node=1</div>
<div> 83387.batch batch node001 03:34:26 OUT_OF_ME+ 0:125 128Gn 1617705K 7880672K cpu=16,mem=0,node=1</div>
83387.extern extern node001 03:34:26 COMPLETED 0:0 128Gn 460K 153196K billing=16,cpu=16,node=1<br>
</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Thanks in advance,</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
Dave</div>
<div style="font-family:"Courier New",monospace;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191x_Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:"Courier New",monospace">
<div><font size="2"><span style="font-size:10pt">
<div></div>
<div style="font-family:"Courier New",monospace;font-size:13.3333px">
</div>
<span id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191x_ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div><a href="mailto:dwc62@drexel.edu" target="_blank">dwc62@drexel.edu</a> 215.571.4335 (o)</div>
<div>For URCF support: <a href="mailto:urcf-support@drexel.edu" target="_blank">urcf-support@drexel.edu</a></div>
<div><a href="https://proteusmaster.urcf.drexel.edu/urcfwiki" target="_blank">https://proteusmaster.urcf.drexel.edu/urcfwiki</a></div>
<div>github:prehensilecode</div>
</div>
<span id="gmail-m_1161703617889707623gmail-m_-2324781006536578760gmail-m_3856910559154608191x_ms-rterangepaste-end"></span>
<div><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<br>
<p align="Left" style="font-family:Calibri;font-size:10pt;color:rgb(0,0,0);margin:5pt">
Drexel Internal Data<br>
</p>
</div>
<br>
<p align="Left" style="font-family:Calibri;font-size:10pt;color:rgb(0,0,0);margin:5pt">
Drexel Internal Data<br>
</p>
</div>
<br>
<p style="font-family:Calibri;font-size:10pt;color:rgb(0,0,0);margin:5pt" align="Left">
Drexel Internal Data<br>
</p>
</div>
</blockquote></div>