<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hello all:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We upgraded from 20.11.8 to 21.08.5 (CentOS 7.9, Slurm built without<o:p></o:p></p>
<p class="MsoNormal">pmix support) recently. After that, we found that in many cases,<o:p></o:p></p>
<p class="MsoNormal">'mpirun' was causing multi-node MPI jobs to have all MPI ranks within<o:p></o:p></p>
<p class="MsoNormal">a node run on the same core. We've moved on to 'srun'.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Now we see a problem in which the OOM killer is in some cases<o:p></o:p></p>
<p class="MsoNormal">predictably killing job steps who don't seem to deserve it. In some<o:p></o:p></p>
<p class="MsoNormal">cases these are job scripts and input files which ran fine before our<o:p></o:p></p>
<p class="MsoNormal">Slurm upgrade. More details follow, but that's it the issue in a<o:p></o:p></p>
<p class="MsoNormal">nutshell.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Other than the version, our one Slurm config change was to remove the<o:p></o:p></p>
<p class="MsoNormal">deprecated 'TaskPluginParam=Sched' from slurm.conf, giving it its<o:p></o:p></p>
<p class="MsoNormal">default 'null' value. Our TaskPlugin remains<o:p></o:p></p>
<p class="MsoNormal">'task/affinity,task/cgroup'.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We've had apparently correct cgroup-based mem limit enforcement in<o:p></o:p></p>
<p class="MsoNormal">place for a long time, so the OOM-killing of the jobs I’m referencing is a<o:p></o:p></p>
<p class="MsoNormal">change in behavior.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Below are some of our support team's findings. I haven't finished<o:p></o:p></p>
<p class="MsoNormal">trying to correlate the anomalous job events with specific OOM<o:p></o:p></p>
<p class="MsoNormal">complaints, or recorded job resource usage at those times. I'm just<o:p></o:p></p>
<p class="MsoNormal">throwing out this message in case what we've seen so far, or the<o:p></o:p></p>
<p class="MsoNormal">painfully obvious thing I’m missing, looks familiar to anyone. Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Application: VASP 6.1.2 launched with srun<o:p></o:p></p>
<p class="MsoNormal">MPI libraries: intel/2019b<o:p></o:p></p>
<p class="MsoNormal">Observations:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Test 1. QDR-fabric Intel nodes (20 nodes x 10 cores/node) outcome:<o:p></o:p></p>
<p class="MsoNormal"> job failed right away, no output generated error text: 20<o:p></o:p></p>
<p class="MsoNormal"> occurrences resembling in form "[13:ra8-10] unexpected reject<o:p></o:p></p>
<p class="MsoNormal"> event from 9:ra8-9"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Test 2. EDR-fabric Intel nodes (20 nodes x 10 cores/node)<o:p></o:p></p>
<p class="MsoNormal"> outcome: job ran for 12 minutes, generated some output data that look fine<o:p></o:p></p>
<p class="MsoNormal"> error text: no error messages, job failed.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Test 3. AMD Rome (20 nodes x 10 cores/node)<o:p></o:p></p>
<p class="MsoNormal"> outcome: job completed successfully after 31 minutes, user<o:p></o:p></p>
<p class="MsoNormal"> confirmed the results are fine<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Application: Quantum Espresso 6.5 launched with srun<o:p></o:p></p>
<p class="MsoNormal">MPI libraries: intel/2019b<o:p></o:p></p>
<p class="MsoNormal">Observations:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- Works correctly when using: 1 node x 64 cores 64 MPI processes), 1x128 (128 MPI processes) (other<o:p></o:p></p>
<p class="MsoNormal"> QE parameters -nk 1 -nt 4 , mem-per-cpu=1500mb)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- A few processes get OOM killed after a while when using: 4 nodes x 32<o:p></o:p></p>
<p class="MsoNormal"> cores (128 MPI processes), 4 nodes x 64 cores (256 MPI processes)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">- job fails within seconds: 16 nodes x 8 cores<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">-- <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Paul Brunk, system administrator</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:black">Georgia Advanced Resource Computing Center<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Enterprise IT Svcs, the University of Georgia<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>