<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi Paul,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 10/02/2022 14:33, Paul Brunk wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BN8PR02MB5954AEC262DECA621606F06DC32F9@BN8PR02MB5954.namprd02.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1"><o:p> </o:p>
<p class="MsoNormal">Now we see a problem in which the OOM
killer is in some cases<o:p></o:p></p>
<p class="MsoNormal">predictably killing job steps who don't
seem to deserve it. In some<o:p></o:p></p>
<p class="MsoNormal">cases these are job scripts and input files
which ran fine before our<o:p></o:p></p>
<p class="MsoNormal">Slurm upgrade. More details follow, but
that's it the issue in a<o:p></o:p></p>
<p class="MsoNormal">nutshell.</p>
</div>
</blockquote>
<p>I'm not sure if this is the case but it might help to keep in
mind the difference between mpirun and srun.</p>
<p>With srun you let slurm create tasks with the appropriate mem/cpu
etc limits and the mpi ranks will run directly in a task.<br>
</p>
<p>With mpirun you usually let your MPI distribution start on task
per node which will spawn the mpi manager which will start the
actual mpi program.</p>
<p>You might very well end up with different memory limits per
process which could be the cause of your OOM issue. Especially if
not all MPI ranks use the same amount of memory.<br>
</p>
<p>Ward<br>
</p>
</body>
</html>