<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>We also noticed the same thing with 21.08.5. In the 21.08 series
SchedMD changed the way they handle cgroups to set the stage for
cgroups v2 (see: <a class="moz-txt-link-freetext" href="https://slurm.schedmd.com/SLUG21/Roadmap.pdf">https://slurm.schedmd.com/SLUG21/Roadmap.pdf</a>).
The 21.08.5 introduced a bug fix which then caused mpirun to not
pin properly (particularly for older versions of MPI):
<a class="moz-txt-link-freetext" href="https://github.com/SchedMD/slurm/blob/slurm-21-08-5-1/NEWS">https://github.com/SchedMD/slurm/blob/slurm-21-08-5-1/NEWS</a> What
we've recommended to users who have hit this was to swap over to
using srun instead of mpirun and the situation clears up.</p>
<p>-Paul Edmon-<br>
</p>
<div class="moz-cite-prefix">On 2/10/2022 8:59 AM, Ward Poelmans
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:b903268d-b3b2-94b5-e9a0-540cf17791d3@vub.be">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">Hi Paul,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 10/02/2022 14:33, Paul Brunk
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN8PR02MB5954AEC262DECA621606F06DC32F9@BN8PR02MB5954.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1"><o:p> </o:p>
<p class="MsoNormal">Now we see a problem in which the OOM
killer is in some cases<o:p></o:p></p>
<p class="MsoNormal">predictably killing job steps who don't
seem to deserve it. In some<o:p></o:p></p>
<p class="MsoNormal">cases these are job scripts and input
files which ran fine before our<o:p></o:p></p>
<p class="MsoNormal">Slurm upgrade. More details follow, but
that's it the issue in a<o:p></o:p></p>
<p class="MsoNormal">nutshell.</p>
</div>
</blockquote>
<p>I'm not sure if this is the case but it might help to keep in
mind the difference between mpirun and srun.</p>
<p>With srun you let slurm create tasks with the appropriate
mem/cpu etc limits and the mpi ranks will run directly in a
task.<br>
</p>
<p>With mpirun you usually let your MPI distribution start on task
per node which will spawn the mpi manager which will start the
actual mpi program.</p>
<p>You might very well end up with different memory limits per
process which could be the cause of your OOM issue. Especially
if not all MPI ranks use the same amount of memory.<br>
</p>
<p>Ward<br>
</p>
</blockquote>
</body>
</html>