<div dir="ltr">Mark,<div><br><div>Thanks for responding.</div><div><br></div><div>Yes, it will constrain it to the amount of memory the user asked for. In fact I have gotten that to work. </div><div><br></div><div>That is not the behavior that we desire (at least initially). The test code I ran through (which just allocates chunks of RAM in a loop) would be <b>constrained </b>to the amount of RAM I asked for but it would not die because it exceeded. (Which make sense because it is being contrained). It would continue to run and I would observe the RSS size grow to 100% of what the allocation I requested, then decrease to about 75% of the allocation, increase back to 100% then decrease again etc. etc. </div><div><br></div><div>Our users have to continually try various memory allocations for the jobs they run and they would prefer that they job die when it runs out of memory, rather than just hang. </div><div><br></div><div>Also, the Slurm Release notes for 19.05 say that cgroup and NoOverMemoryKill are not compatible:</div><div><br></div><div><a href="https://slurm.schedmd.com/news.html">https://slurm.schedmd.com/news.html</a><br></div><div><br></div><div>NOTE: slurmd and slurmctld will now fatal if two incompatible mechanisms for<br> enforcing memory limits are set. <span style="background-color:rgb(255,255,0)">This makes incompatible the use of<br> task/cgroup memory limit enforcing (Constrain[RAM|Swap]Space=yes) with<br> JobAcctGatherParams=OverMemoryKill,</span> which could cause problems when a<br> task is killed by one of them while the other is at the same time<br> managing that task. The NoOverMemoryKill setting has been deprecated in<br> favor of OverMemoryKill, since now the default is *NOT* to have any<br> memory enforcement mechanism.<br></div><div><br></div><div>Have you gotten this to work with 19.0.5?</div></div><div><br></div><div>Thanks Mike</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Oct 25, 2019 at 9:41 AM Mark Hahn <<a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> need. I simply want to enforce the memory limits as specified by the user<br>
> at job submission time. This seems to have been the behavior in previous<br>
<br>
but cgroups (with Constrain) do that all by themselves.<br>
<br>
> If someone could post just a simple slurm.conf file that forces the memory<br>
> limits to be honored (and kills the job if they are exceeded), then I could<br>
> extract what I need from that.<br>
<br>
slurm.conf:<br>
TaskPlugin=task/cgroup<br>
JobAcctGatherParams=NoOverMemoryKill<br>
<br>
cgroup.conf:<br>
ConstrainRAMSpace=yes<br>
ConstrainKmemSpace=no<br>
ConstrainSwapSpace=yes<br>
AllowedRamSpace=100<br>
AllowedSwapSpace=0<br>
MaxRAMPercent=100<br>
MaxSwapPercent=100<br>
MinRAMSpace=30<br>
<br>
I think those are the relevant portions.<br>
<br>
regards,<br>
-- <br>
operator may differ from spokesperson. <a href="mailto:hahn@mcmaster.ca" target="_blank">hahn@mcmaster.ca</a><br>
<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px"><div dir="ltr"><div><span style="font-family:"times new roman",serif"><b><i>J. Michael Mosley</i></b><br>University Research Computing<br>The University of North Carolina at Charlotte<br>9201 University City Blvd<br>Charlotte, NC 28223<br><u>704.687.7065 </u> <u> j<i><a href="mailto:mmosley@uncc.edu" target="_blank">mmosley@uncc.edu</a></i></u></span></div></div></div></div></div>