[slurm-users] OverMemoryKill Not Working?

Fri Oct 25 17:22:09 UTC 2019

Jurgen,

Thank you for all of the information.  I appreciate you taking the time to
test the configuration with 19.05.
I feel a little better about my efforts now. :-)

I will check out some of your suggestions to mitigate the issue.
Ultimately, we will probably use cgroups for containment but as I explained
in my earlier post, we wanted to have the kill option available initially
and it seemed like it should have been simple to set up.

It would be interesting to know if anyone else could comment as to 19.05
seeming to act differently than previous versions in light of  the release
notes/ documentation.

Again, thank your time.   It was very helpful.

Mike

On Fri, Oct 25, 2019 at 11:26 AM Juergen Salk <juergen.salk at uni-ulm.de>
wrote:

> Hi Mike,
>
> IIRC, I once did some tests with the very same configuration as
> your's, i.e. `JobAcctGatherType=jobacct_gather/linux´ and
> `JobAcctGatherParams=OverMemoryKill´ and got this to work as expected:
> Jobs were killed when they exceeded the requested amount of memory.
> This was with Slurm 18.08.7. After some tests I went back
> to memory enforcement with cgroups as this also keeps memory into
> account that is consumed by writing data to a tmpfs filesystem, such
> as /dev/shm.
>
> I have now restored the old configuration, that I think I've used to
> experiment with Slurm's capabilities to enforce memory usage without
> cgroups and then tried again in my test cluster (now running Slurm
> 19.05.2). As far as I understand, the configuration above should
> also work with 19.05.2.
>
> But I was surprised to see, that I can also reproduce the
> behavior that you described: A process that exceeds the
> requested amount of memory keeps happily running.
>
> Anyway, I think memory enforcement with cgroups is more reliable
> and, thus, more commonly used these days. Recently there was an
> interesting discussion on this list about how to get Slurm to cancel
> the whole job if the memory is exceeded (not just oom-kill some
> processes). Someone suggested setting `KillOnBadExit=1´ in
> slurm.conf. Someone else suggested using `set -o errexit´ (or
> #!/bin/bash -e instead of plain #!/bin/bash) in the job scripts, so
> that the failure of any command within the script will cause the job
> to stop immediately.
>
> You may find the thread in the list archive if you search for
> "How to automatically kill a job that exceeds its memory limits".
>
> Best regards
> Jürgen
>
> --
> Jürgen Salk
> Scientific Software & Compute Services (SSCS)
> Kommunikations- und Informationszentrum (kiz)
> Universität Ulm
> Telefon: +49 (0)731 50-22478
> Telefax: +49 (0)731 50-22471
>
>
>
> * Mike Mosley <Mike.Mosley at uncc.edu> [191025 09:17]:
> > Ahmet,
> >
> > Thank you for taking the time to respond to my question.
> >
> > Yes, the --mem=1GBB is a typo.   It's correct in my script, I just
> > fat-fingered it in the email. :-)
> >
> > BTW, the exact version I am using is 19.05.*2.*
> >
> > Regarding your response, it seems that that might be more than what I
> > need.   I simply want to enforce the memory limits as specified by the
> user
> > at job submission time.   This seems to have been the behavior in
> previous
> > versions of Slurm.   What I want is what is described in the 19.05
> release
> > notes:
> >
> >
> >
> > *RELEASE NOTES FOR SLURM VERSION 19.0528 May 2019*
> >
> >
> >
> > *NOTE: slurmd and slurmctld will now fatal if two incompatible mechanisms
> > for      enforcing memory limits are set. This makes incompatible the use
> > of      task/cgroup memory limit enforcing (Constrain[RAM|Swap]Space=yes)
> > with      JobAcctGatherParams=OverMemoryKill, which could cause problems
> > when a      task is killed by one of them while the other is at the same
> > time      managing that task. The NoOverMemoryKill setting has been
> > deprecated in      favor of OverMemoryKill, since now the default is
> *NOT*
> > to have any      memory enforcement mechanism.NOTE: MemLimitEnforce
> > parameter has been removed and the functionality that      was provided
> > with it has been merged into a JobAcctGatherParams. It      may be
> enabled
> > by setting JobAcctGatherParams=OverMemoryKill, so now      job and steps
> > killing by OOM is enabled from the same place.*
> >
> >
> >
> > So, is it really necessary to do what you suggested to get that
> > functionality?
> >
> > If someone could post just a simple slurm.conf file that forces the
> memory
> > limits to be honored (and kills the job if they are exceeded), then I
> could
> > extract what I need from that.
> >
> > Again, thanks for the assistance.
> >
> > Mike
> >
> >
> >
> > On Thu, Oct 24, 2019 at 11:27 PM mercan <ahmet.mercan at uhem.itu.edu.tr>
> > wrote:
> >
> > > Hi;
> > >
> > > You should set
> > >
> > > SelectType=select/cons_res
> > >
> > > and plus one of these:
> > >
> > > SelectTypeParameters=CR_Memory
> > > SelectTypeParameters=CR_Core_Memory
> > > SelectTypeParameters=CR_CPU_Memory
> > > SelectTypeParameters=CR_Socket_Memory
> > >
> > > to open Memory allocation tracking according to documentation:
> > >
> > > https://slurm.schedmd.com/cons_res_share.html
> > >
> > > Also, the line:
> > >
> > > #SBATCH --mem=1GBB
> > >
> > > contains "1GBB". Is this same at job script?
> > >
> > >
> > > Regards;
> > >
> > > Ahmet M.
> > >
> > >
> > > 24.10.2019 23:00 tarihinde Mike Mosley yazdı:
> > > > Hello,
> > > >
> > > > We are testing Slurm19.05 on Linux RHEL7.5+ with the intent to
> migrate
> > > > from it toTorque/Moab in the near future.
> > > >
> > > > One of the things our users are used to is that when their jobs
> exceed
> > > > the amount of memory they requested, the job is terminated by the
> > > > scheduler.   We realize the Slurm prefers to use cgroups to contain
> > > > rather than kill the jobs but initially we need to have the kill
> > > > option in place to transition our users.
> > > >
> > > > So, looking at the documentation, it appears that in 19.05, the
> > > > following needs to be set to accomplish this:
> > > >
> > > > JobAcctGatherParams = OverMemoryKill
> > > >
> > > >
> > > > Other possibly relevant settings we made:
> > > >
> > > > JobAcctGatherType = jobacct_gather/linux
> > > >
> > > > ProctrackType = proctrack/linuxproc
> > > >
> > > >
> > > > We have avoided configuring any cgroup parameters for the time being.
> > > >
> > > > Unfortunately, when we submit a job with the following:
> > > >
> > > > #SBATCH --nodes=1
> > > >
> > > > #SBATCH --ntasks-per-node=1
> > > >
> > > > #SBATCH --mem=1GBB
> > > >
> > > >
> > > > We see RSS ofthe  job steadily increase beyond the 1GB limit and it
> is
> > > > never killed.    Interestingly enough, the proc information shows the
> > > > ulimit (hard and soft) for the process set to around 1GB.
> > > >
> > > > We have tried various settings without any success.   Can anyone
> point
> > > > out what we are doing wrong?
> > > >
> > > > Thanks,
> > > >
> > > > Mike
> > > >
> > > > --
> > > > */J. Michael Mosley/*
> > > > University Research Computing
> > > > The University of North Carolina at Charlotte
> > > > 9201 University City Blvd
> > > > Charlotte, NC  28223
> > > > _704.687.7065 _ _ j/mmosley at uncc.edu <mailto:mmosley at uncc.edu>/_
> > >
> >
> >
> > --
> > *J. Michael Mosley*
> > University Research Computing
> > The University of North Carolina at Charlotte
> > 9201 University City Blvd
> > Charlotte, NC  28223
> > *704.687.7065 *    * jmmosley at uncc.edu <mmosley at uncc.edu>*
>
>
>
> --
> GPG A997BA7A | 87FC DA31 5F00 C885 0DC3  E28F BD0D 4B33 A997 BA7A
>
>

-- 
*J. Michael Mosley*
University Research Computing
The University of North Carolina at Charlotte
9201 University City Blvd
Charlotte, NC  28223
*704.687.7065 *    * jmmosley at uncc.edu <mmosley at uncc.edu>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20191025/f925d89b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5329 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20191025/f925d89b/attachment.bin>