<div dir="ltr">Hi Roger,<div><br></div><div>Thanks for the response. I am pretty sure the job is actually getting killed. I don't see it running in the process table and the local SLURM log just displays:</div><div><br></div><div><font face="monospace">[2021-08-10T16:31:36.139] [6628753.batch] error: Detected 1 oom-kill event(s) in StepId=6628753.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.</font><br></div><div><br></div><div>Best,</div><div><br>Sean</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 10, 2021 at 5:13 PM Roger Moye <<a href="mailto:rmoye@quantlab.com">rmoye@quantlab.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_3556646625717157748WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Do you know if the job is actually being killed? We had an issue on an older version of slurm whereby we got OOM errors but the tasks actually completed. The
OOM came when the job exited and was a false error.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Also, there are several bug reports open right now about an issue similar to what you have described. You can go to <a href="http://bugs.schedmd.com" target="_blank">bugs.schedmd.com</a> to look at those bug reports.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">-Roger
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>Sean Caron<br>
<b>Sent:</b> Tuesday, August 10, 2021 4:01 PM<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>>; Sean Caron <<a href="mailto:scaron@umich.edu" target="_blank">scaron@umich.edu</a>><br>
<b>Subject:</b> [slurm-users] Spurious OOM-kills with cgroups on 20.11.8?<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Has anyone else observed jobs getting OOM-killed in 20.11.8 with cgroups that ran fine in previous versions like 20.10?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've had a few reports from users after upgrading maybe six weeks ago that their jobs are getting OOM-killed when they haven't changed anything and the job ran to completion in the past with the same memory specification.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The most recent report I received today involved a job running a "cp" command getting OOM-killed. I have a hard time believing "cp" uses very much memory...<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">These machines are running various 5.4.x or 5.3.x Linux kernels.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've had really good luck with the cgroups OOM-killer the last few years from keeping my nodes getting overwhelmed by runaway jobs. I'd hate to have to disable it just to clean up these weird issues.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">My cgroup.conf file looks like the following:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">CgroupAutomount=yes<br>
<br>
ConstrainCores=yes<br>
<br>
ConstrainRAMSpace=yes<br>
ConstrainSwapSpace=yes<br>
<br>
AllowedRamSpace=100<br>
AllowedSwapSpace=0</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Should I maybe bump AllowedRamSpace? I don't see how this is any different than just asking the user to re-run the job with a larger memory allocation request. And that doesn't explain why jobs suddenly need more memory before getting OOM-killed
than they used to.<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Sean<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p><font style="font-family:Verdana;font-size:11px"><span style="font-family:Verdana"> </span>-----------------------------------------------------------------------------------</font></p><font style="font-family:Verdana;font-size:11px">
</font><p><font style="font-family:Verdana;font-size:11px">The information in this communication and any attachment is confidential and intended solely for the attention and use of the named addressee(s). All information and opinions expressed herein are subject to change without notice. This communication is not to be construed as an offer to sell or the solicitation of an offer to buy any security. Any such offer or solicitation can only be made by means of the delivery of a confidential private offering memorandum (which should be carefully reviewed for a complete description of investment strategies and risks). Any reliance one may place on the accuracy or validity of this information is at their own risk. Past performance is not necessarily indicative of the future results of an investment. All figures are estimated and unaudited unless otherwise noted. If you are not the intended recipient, or a person responsible for delivering this to the intended recipient, you are not authorized to and must not disclose, copy, distribute, or retain this message or any part of it. In this case, please notify the sender immediately at 713-333-5440</font></p></div>
</blockquote></div>