<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 56.7pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Any reason *not* to create an array of 100k jobs and let the scheduler just handle things? Current versions of Slurm support arrays of up to 4M jobs, and you can limit the number of jobs running simultaneously with the '%' specifier in
your array= sbatch parameter.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Arthur Gilly <arthur.gilly@helmholtz-muenchen.de><br>
<b>Date: </b>Tuesday, June 8, 2021 at 4:12 AM<br>
<b>To: </b>'Slurm User Community List' <slurm-users@lists.schedmd.com><br>
<b>Subject: </b>Re: [slurm-users] Kill job when child process gets OOM-killed<o:p></o:p></span></p>
</div>
<p align="center" style="margin:0in;text-align:center;background:white"><b><span style="font-size:12.0pt;color:red;background:white">External Email Warning</span></b><o:p></o:p></p>
<p align="center" style="mso-margin-top-alt:0in;margin-right:12.0pt;margin-bottom:0in;margin-left:12.0pt;text-align:center;background:white">
<b><span style="font-size:12.0pt;color:red">This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.</span></b><o:p></o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="0" width="100%" align="center">
</div>
<div>
<p class="MsoNormal">Thank you Loris!<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Like many of our jobs, this is an embarrassingly parallel analysis, where we have to strike a compromise between what would be a completely granular array of >100,000 small jobs or some kind of serialisation through loops. So the individual
jobs where I noticed this behaviour are actually already part of an array :)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"><br>
Arthur<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">-------------------------------------------------------------<o:p></o:p></p>
<p class="MsoNormal">Dr. Arthur Gilly<o:p></o:p></p>
<p class="MsoNormal">Head of Analytics<o:p></o:p></p>
<p class="MsoNormal">Institute of Translational Genomics<o:p></o:p></p>
<p class="MsoNormal">Helmholtz-Centre Munich (HMGU)<o:p></o:p></p>
<p class="MsoNormal">-------------------------------------------------------------<o:p></o:p></p>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com>
<b>On Behalf Of </b>Loris Bennett<br>
<b>Sent:</b> Tuesday, 8 June 2021 16:05<br>
<b>To:</b> Slurm User Community List <slurm-users@lists.schedmd.com><br>
<b>Subject:</b> Re: [slurm-users] Kill job when child process gets OOM-killed<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Dear Arthur,<br>
<br>
Arthur Gilly <<a href="mailto:arthur.gilly@helmholtz-muenchen.de">arthur.gilly@helmholtz-muenchen.de</a>> writes:<br>
<br>
> Dear Slurm users,<br>
><br>
> <br>
><br>
> I am looking for a SLURM setting that will kill a job immediately when any subprocess of that job hits an OOM limit. Several posts have touched upon that, e.g:
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fslurm-users%40lists.schedmd.com%2Fmsg04091.html&data=04%7C01%7Crenfro%40tntech.edu%7Cfb729b1d5cea45e7d86208d92a5d8021%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637587403358417389%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=uV5WOedg5AGZEGtgz%2FfCmXgr%2BMLBu0rFStpC4c2xt0s%3D&reserved=0">
https://www.mail-archive.com/slurm-users@lists.schedmd.com/msg04091.html</a> and<br>
> <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fslurm-users%40lists.schedmd.com%2Fmsg04190.html&data=04%7C01%7Crenfro%40tntech.edu%7Cfb729b1d5cea45e7d86208d92a5d8021%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637587403358427383%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=YMHuN3gTXlQYR2o0gjysgSqta42AxGs9X3vnuJRpNe4%3D&reserved=0">
https://www.mail-archive.com/slurm-users@lists.schedmd.com/msg04190.html</a> or <a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.schedmd.com%2Fshow_bug.cgi%3Fid%3D3216&data=04%7C01%7Crenfro%40tntech.edu%7Cfb729b1d5cea45e7d86208d92a5d8021%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637587403358427383%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=Zokdp6EumQGD2xQD3KfRvzOGtzwtOjFC4WEyAJjkO%2Bc%3D&reserved=0">
https://bugs.schedmd.com/show_bug.cgi?id=3216</a> but I cannot find an answer that works in our setting.<br>
><br>
> <br>
><br>
> The two options I have found are:<br>
><br>
> 1 Set shebang to #!/bin/bash -e, which we don’t want to do as we’d need to change this for hundreds of scripts from another cluster where we had a different scheduler, AND it would kill tasks for other runtime errors (e.g. if one command in the<br>
> script doesn’t find a file).<br>
><br>
> 2 Set KillOnBadExit=1. I am puzzled by this one. This is supposed to be overridden by srun’s -K option. Using the example below, srun -K --mem=1G ./multalloc.sh would be expected to kill the job at the first OOM. But it doesn’t, and happily<br>
> keeps reporting 3 oom-kill events. So, will this work?<br>
><br>
> <br>
><br>
> The reason we want this is that we have script that execute programs in loops. These programs are slow and memory intensive. When the first one crashes for OOM, the next iterations also crash. In the current setup, we are wasting days<br>
> executing loops where every iteration crashes after an hour or so due to OOM.<br>
<br>
Not an answer to your question, but if your runs are independent, would<br>
using a job array help you here?<br>
<br>
Cheers,<br>
<br>
Loris<br>
<br>
> We are using cgroups (and we want to keep them) with the following config:<br>
><br>
> CgroupAutomount=yes<br>
><br>
> ConstrainCores=yes<br>
><br>
> ConstrainDevices=yes<br>
><br>
> ConstrainKmemSpace=no<br>
><br>
> ConstrainRAMSpace=yes<br>
><br>
> ConstrainSwapSpace=yes<br>
><br>
> MaxSwapPercent=10<br>
><br>
> TaskAffinity=no<br>
><br>
> <br>
><br>
> Relevant bits from slurm.conf:<br>
><br>
> SelectTypeParameters=CR_Core_Memory,CR_ONE_TASK_PER_CORE<br>
><br>
> SelectType=select/cons_tres<br>
><br>
> GresTypes=gpu,mps,bandwidth<br>
><br>
> <br>
><br>
> <br>
><br>
> Very simple example:<br>
><br>
> #!/bin/bash<br>
><br>
> # multalloc.sh – each line is a very simple cpp program that allocates a 8Gb vector and fills it with random floats<br>
><br>
> echo one<br>
><br>
> ./alloc8Gb<br>
><br>
> echo two<br>
><br>
> ./alloc8Gb<br>
><br>
> echo three<br>
><br>
> ./alloc8Gb<br>
><br>
> echo done.<br>
><br>
> <br>
><br>
> This is submitted as follows:<br>
><br>
> <br>
><br>
> sbatch --mem=1G ./multalloc.sh<br>
><br>
> <br>
><br>
> The log is :<br>
><br>
> one<br>
><br>
> ./multalloc.sh: line 4: 231155 Killed ./alloc8Gb<br>
><br>
> two<br>
><br>
> ./multalloc.sh: line 6: 231181 Killed ./alloc8Gb<br>
><br>
> three<br>
><br>
> ./multalloc.sh: line 8: 231263 Killed ./alloc8Gb<br>
><br>
> done.<br>
><br>
> slurmstepd: error: Detected 3 oom-kill event(s) in StepId=3130111.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.<br>
><br>
> <br>
><br>
> I am expecting an OOM job kill right before “two”.<br>
><br>
> <br>
><br>
> Any help appreciated.<br>
><br>
> <br>
><br>
> Best regards,<br>
><br>
> <br>
><br>
> Arthur<br>
><br>
> <br>
><br>
> <br>
><br>
> -------------------------------------------------------------<br>
><br>
> Dr. Arthur Gilly<br>
><br>
> Head of Analytics<br>
><br>
> Institute of Translational Genomics<br>
><br>
> Helmholtz-Centre Munich (HMGU)<br>
><br>
> -------------------------------------------------------------<br>
><br>
> <br>
><br>
> Helmholtz Zentrum München <br>
> Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH) <br>
> Ingolstädter Landstr. 1 <br>
> 85764 Neuherberg <br>
> <a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.helmholtz-muenchen.de%2F&data=04%7C01%7Crenfro%40tntech.edu%7Cfb729b1d5cea45e7d86208d92a5d8021%7C66fecaf83dc04d2cb8b8eff0ddea46f0%7C1%7C0%7C637587403358437375%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=vO%2FcbBB%2BpvnYoPQrGmeANVvgbwzIidUXdKV2%2FTmH3jI%3D&reserved=0">
www.helmholtz-muenchen.de</a> <br>
> Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling <br>
> Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Kerstin Günther<br>
> Registergericht: Amtsgericht München HRB 6466 <br>
> USt-IdNr: DE 129521671 <br>
><br>
-- <br>
Dr. Loris Bennett (Hr./Mr.)<br>
ZEDAT, Freie Universität Berlin Email <a href="mailto:loris.bennett@fu-berlin.de">
loris.bennett@fu-berlin.de</a><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
Helmholtz Zentrum München <br>
Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH) <br>
Ingolstädter Landstr. 1 <br>
85764 Neuherberg <br>
www.helmholtz-muenchen.de <br>
Aufsichtsratsvorsitzende: MinDir.in Prof. Dr. Veronika von Messling <br>
Geschäftsführung: Prof. Dr. med. Dr. h.c. Matthias Tschöp, Kerstin Günther<br>
Registergericht: Amtsgericht München HRB 6466 <br>
USt-IdNr: DE 129521671 <br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
</body>
</html>