<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hello,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thank you all for your useful replies. I double checked that the oom-killer "fires" at the end of every job on our cluster. As you mention this isn't significant and not something to be concerned about. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Best regards,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
David</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Marcus Wagner <wagner@itc.rwth-aachen.de><br>
<b>Sent:</b> 08 November 2019 13:00<br>
<b>To:</b> slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] oom-kill events for no good reason</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">Hi David,<br>
<br>
yes, I see these messages also. I also think, this is more likely a wrong message. If a job has been cancelled by the OOM-Killer, you can see this with sacct, e.g.<br>
$> sacct -j 10816098<br>
JobID JobName Partition Account AllocCPUS State ExitCode <br>
------------ ---------- ---------- ---------- ---------- ---------- -------- <br>
10816098 VASP_MPI c18m default 12 OUT_OF_ME+ 0:125 <br>
10816098.ba+ batch default 12 OUT_OF_ME+ 0:125 <br>
10816098.ex+ extern default 12 COMPLETED 0:0 <br>
10816098.0 vasp_mpi default 12 OUT_OF_ME+ 0:125 <br>
<br>
Best<br>
Marcus<br>
<br>
<div class="x_moz-cite-prefix">On 11/7/19 5:36 PM, David Baker wrote:<br>
</div>
<blockquote type="cite"><style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hello, </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
We are dealing with some weird issue on our shared nodes where job appear to be stalling for some reason. I was advised that this issue might be related to the oom-killer process. We do see a lot of these events. In fact when I started to take a closer look
this afternoon I noticed that all jobs on all nodes (not just the shared nodes) are "firing" the oom-killer for some reason when they finish. </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
As a demo I launched a very simple (low memory usage) test jobs on a shared node and then after a few minutes cancelled it to show the behaviour. Looking in the slurmd.log -- see below -- we see the oom-killer being fired for no good reason. This "feels" vaguely
similar to this bug -- <a href="https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D5121&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb280bfbe58bb495bbace08d7644c9e52%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=g%2BT6zIZqTr8ZAi52RgFRaMViwdxZPjkEOkvNa6YEXRU%3D&reserved=0" originalsrc="https://bugs.schedmd.com/show_bug.cgi?id=5121" shash="BklY9bDehJk5r1rV2SqeM5XU12rtQvgZxmOCp/z4mfN/eTL+wW//sZEA6tZpQP3gE+3HTmuNg5bevoSnC1eYOD+b+Y4aGTjca1mDuHQzt2gTgvvjaN4F9jsAo9iHGqJAYJPtLLb9H+Maz3wHoBBewTMLnHYfASZuFDuUQW7EX1k=">https://bugs.schedmd.com/show_bug.cgi?id=5121</a> which
I understand was patched back in SLURM v17 (we are using v18*). </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Has anyone else seen this behaviour? Or more to the point does anyone understand this behaviour and know how to squash it, please?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Best regards,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
David</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>[2019-11-07T16:14:52.551] Launching batch job 164978 for UID 57337<br>
</span>
<div>[2019-11-07T16:14:52.559] [164977.batch] task/cgroup: /slurm/uid_57337/job_164977: alloc=23640MB mem.limit=23640MB memsw.limit=unlimited<br>
</div>
<div>[2019-11-07T16:14:52.560] [164977.batch] task/cgroup: /slurm/uid_57337/job_164977/step_batch: alloc=23640MB mem.limit=23640MB memsw.limit=unlimited<br>
</div>
<div>[2019-11-07T16:14:52.584] [164978.batch] task/cgroup: /slurm/uid_57337/job_164978: alloc=23640MB mem.limit=23640MB memsw.limit=unlimited<br>
</div>
<div>[2019-11-07T16:14:52.584] [164978.batch] task/cgroup: /slurm/uid_57337/job_164978/step_batch: alloc=23640MB mem.limit=23640MB memsw.limit=unlimited<br>
</div>
<div>[2019-11-07T16:14:52.960] [164977.batch] task_p_pre_launch: Using sched_affinity for tasks<br>
</div>
<div>[2019-11-07T16:14:52.960] [164978.batch] task_p_pre_launch: Using sched_affinity for tasks<br>
</div>
<div>[2019-11-07T16:16:05.859] [164977.batch] error: *** JOB 164977 ON gold57 CANCELLED AT 2019-11-07T16:16:05 ***<br>
</div>
<div>[2019-11-07T16:16:05.882] [164977.extern] <b>_oom_event_monitor: oom-kill event count: 1</b><br>
</div>
<span>[2019-11-07T16:16:05.886] [164977.extern] done with job</span><br>
</div>
</blockquote>
<br>
<pre class="x_moz-signature" cols="72">--
Marcus Wagner, Dipl.-Inf.
IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
<a class="x_moz-txt-link-abbreviated" href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>
<a class="x_moz-txt-link-abbreviated" href="https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.itc.rwth-aachen.de&data=01%7C01%7Cd.j.baker%40soton.ac.uk%7Cb280bfbe58bb495bbace08d7644c9e52%7C4a5378f929f44d3ebe89669d03ada9d8%7C0&sdata=%2Bk3%2BvCTzz%2ByeelQ96SPB5N0EoXCtWp0mrX9pFrUsHHk%3D&reserved=0" originalsrc="http://www.itc.rwth-aachen.de" shash="A65zYoOiClXSS21Visq7ZLOHGzUXky/3TUIIUExRR9LrmfT1dBgJ/Yx31X0ZrHcd82UNOfWBmI4KwvgdzpE3CYKfQbYFYhhKcp8QnrogMWmBLXygAQ7pQHbA5I30Ib3cnBBCPn0w4MRLYzcqUT7zJjqXLeIzIKq4bqkjUhGhsZo=">www.itc.rwth-aachen.de</a>
</pre>
</div>
</body>
</html>