<div dir="ltr"><div>Hi Marcus,</div><div><br></div><div>That makes sense, thanks! I suppose then (for monitoring purposes, for example, without probing scontrol/sacct) if you wanted to figure out the true maximum memory limit for a task, you'd need to walk up the hierarchy and take whatever the smallest value you find is.</div><div><br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="font-family:arial,helvetica,sans-serif"><span style="color:rgb(7,55,99)"><font size="2">____</font></span></span><span style="color:rgb(7,55,99)"><font size="2"><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">_____</span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span></span></span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif"><span><span style="font-family:arial,helvetica,sans-serif">__________</span></span></span></span></span></span>_____</span></span>______<br><b>Jacob D. Chappell, CSM</b><br>Research Computing | Research Computing Infrastructure<br>Information Technology Services | University of Kentucky<br><a href="mailto:jacob.chappell@uky.edu" target="_blank">jacob.chappell@uky.edu</a><br></span></font></span></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jun 23, 2021 at 6:32 AM Marcus Wagner <<a href="mailto:wagner@itc.rwth-aachen.de">wagner@itc.rwth-aachen.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">CAUTION: External Sender<br>
<br>
<br><br><br>---------- Forwarded message ----------<br>From: Marcus Wagner <<a href="mailto:wagner@itc.rwth-aachen.de" target="_blank">wagner@itc.rwth-aachen.de</a>><br>To: <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>Cc: <br>Bcc: <br>Date: Wed, 23 Jun 2021 13:30:14 +0200<br>Subject: Re: [slurm-users] Slurm does not set memory.limit_in_bytes for tasks (but does for steps)<br>Hi Jacob,<br>
<br>
I generally think, that that is the better way.<br>
If you have e.g. tasks with different memory needs, Slurm (or the oom_killer to be precise) would kill the job, if that limit gets exceeded. If the limit is set for the step, the tasks can "steal" memory from each other.<br>
<br>
<br>
Best<br>
Marcus<br>
<br>
Am 22.06.2021 um 18:46 schrieb Jacob Chappell:<br>
> Hello everyone,<br>
> <br>
> I came across a weird behavior and was wondering if this is a bug, oversight, or intended?<br>
> <br>
> It appears that Slurm does not set memory.limit_in_bytes at the task level, but it does set it at the step level and above. Observe:<br>
> <br>
> $ grep memory /proc/$$/cgroup<br>
> 10:memory:/slurm/uid_2001/job_304876/step_0/task_0<br>
> <br>
> $ cd /sys/fs/cgroup/memory/slurm/uid_2001/job_304876/step_0/task_0<br>
> <br>
> $ cat memory.limit_in_bytes<br>
> 9223372036854771712     <--- basically unlimited<br>
> <br>
> But lets check the parent:<br>
> <br>
> $ cat ../memory.limit_in_bytes<br>
> 33554432000      <-- set properly to around 32 GB, see below<br>
> <br>
> $ scontrol show job 304876 | grep mem=<br>
>     TRES=cpu=8,mem=*32000M*,node=1,billing=8<br>
> <br>
> Now, it does appear that the task is still limited to the step's memory limit given the hierarchical nature of cgroups, but I just wanted to mention this anyway and see if anyone had any thoughts.<br>
> <br>
> Thanks,<br>
> __________________________________________________<br>
> *Jacob D. Chappell, CSM*<br>
> Research Computing | Research Computing Infrastructure<br>
> Information Technology Services | University of Kentucky<br>
> <a href="mailto:jacob.chappell@uky.edu" target="_blank">jacob.chappell@uky.edu</a> <mailto:<a href="mailto:jacob.chappell@uky.edu" target="_blank">jacob.chappell@uky.edu</a>><br>
<br>
-- <br>
Dipl.-Inf. Marcus Wagner<br>
<br>
IT Center<br>
Gruppe: Systemgruppe Linux<br>
Abteilung: Systeme und Betrieb<br>
RWTH Aachen University<br>
Seffenter Weg 23<br>
52074 Aachen<br>
Tel: +49 241 80-24383<br>
Fax: +49 241 80-624383<br>
<a href="mailto:wagner@itc.rwth-aachen.de" target="_blank">wagner@itc.rwth-aachen.de</a><br>
<a href="http://www.itc.rwth-aachen.de" rel="noreferrer" target="_blank">www.itc.rwth-aachen.de</a><br>
<br>
Social Media Kanäle des IT Centers:<br>
<a href="https://blog.rwth-aachen.de/itc/" rel="noreferrer" target="_blank">https://blog.rwth-aachen.de/itc/</a><br>
<a href="https://www.facebook.com/itcenterrwth" rel="noreferrer" target="_blank">https://www.facebook.com/itcenterrwth</a><br>
<a href="https://www.linkedin.com/company/itcenterrwth" rel="noreferrer" target="_blank">https://www.linkedin.com/company/itcenterrwth</a><br>
<a href="https://twitter.com/ITCenterRWTH" rel="noreferrer" target="_blank">https://twitter.com/ITCenterRWTH</a><br>
<a href="https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ" rel="noreferrer" target="_blank">https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ</a><br>
<br>
</blockquote></div>