<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;">Hi, Sean:</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;">Slurm version 20.02.6 (via Bright Cluster Manager)</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
</div>
<span style="font-family: "Courier New", monospace;"> ProctrackType=proctrack/cgroup</span>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;"> JobAcctGatherType=jobacct_gather/linux</span><br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;"> JobAcctGatherParams=UsePss,NoShared</span><br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;">I just skimmed </span><a href="https://bugs.schedmd.com/show_bug.cgi?id=5549" id="LPlnk"><span style="font-family: "Courier New", monospace;">https://bugs.schedmd.com/show_bug.cgi?id=5549</span></a><span style="font-family: "Courier New", monospace;"> because
this job appeared to have left two slurmstepd zombie processes running at 100%CPU each, and changed to:</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"> ProctrackType=proctrack/cgroup</span>
<div style="margin:0px;font-size:12pt"><span style="font-family: "Courier New", monospace;"> JobAcctGatherType=jobacct_gather/cgroup</span><br>
</div>
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"> JobAcctGatherParams=UsePss,NoShared,NoOverMemoryKill</span><br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;">Have asked the user to re-run the job, but that has not happened, yet.</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;">cgroup.conf:</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"> CgroupMountpoint="/sys/fs/cgroup"
<div> CgroupAutomount=yes</div>
<div> TaskAffinity=yes</div>
<div> ConstrainCores=yes</div>
<div> ConstrainRAMSpace=yes</div>
<div> ConstrainSwapSpace=no</div>
<div> ConstrainDevices=yes</div>
<div> ConstrainKmemSpace=yes</div>
<div> AllowedRamSpace=100.00</div>
<div> AllowedSwapSpace=0.00</div>
<div> MinKmemSpace=200</div>
<div> MaxKmemPercent=100.00</div>
<div> MemorySwappiness=100</div>
<div> MaxRAMPercent=100.00</div>
<div> MaxSwapPercent=100.00</div>
MinRAMSpace=200<br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="margin: 0px; font-size: 12pt; font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
Dave</div>
<div>
<div style="font-family: "Courier New", monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div></div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:'Courier New',monospace">
<div class="BodyFragment"><font size="2"><span style="font-size:10pt">
<div class="PlainText"></div>
<div class="PlainText" style="font-family:"Courier New",monospace; font-size:13.3333px">
</div>
<span id="ms-rterangepaste-start"></span>
<div>--</div>
<div>
<div>David Chin, PhD (he/him) Sr. SysAdmin, URCF, Drexel</div>
<div>dwc62@drexel.edu 215.571.4335 (o)</div>
<div>For URCF support: urcf-support@drexel.edu</div>
<div>https://proteusmaster.urcf.drexel.edu/urcfwiki</div>
<div>github:prehensilecode</div>
</div>
<span id="ms-rterangepaste-end"></span>
<div class="PlainText"><br>
</div>
</span></font></div>
</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family:"Courier New",monospace; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Sean Crosby <scrosby@unimelb.edu.au><br>
<b>Sent:</b> Monday, March 15, 2021 15:22<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] [EXT] Job ended with OUT_OF_MEMORY even though MaxRSS and MaxVMSize are under the ReqMem value</font>
<div> </div>
</div>
<div>
<table width="100%">
<tbody>
<tr>
<td style="border-left:4px solid goldenrod; background:cornsilk; padding:0 3pt">
<p style="font:small-caps bold 100% sans-serif">External.</p>
</td>
</tr>
</tbody>
</table>
<div>
<div dir="ltr">
<div>What are your Slurm settings - what's the values of</div>
<div><br>
</div>
ProctrackType<br>
JobAcctGatherType<br>
JobAcctGatherParams<br>
<br>
<div>and what's the contents of cgroup.conf? Also, what version of Slurm are you using?<br>
</div>
<div><br>
</div>
<div>Sean</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="x_gmail_signature">--<br>
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead<br>
Research Computing Services | Business Services<br>
The University of Melbourne, Victoria 3010 Australia<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<p style="font-family:Calibri;font-size:10pt;color:#000000;margin:5pt;" align="Left">
Drexel Internal Data<br>
</p>
</body>
</html>