[slurm-users] Gentle memory limits in Slurm using cgroup?

[Janne Blomqvist]

On 02/05/2019 17.53, Alexander Åhman wrote:
> Hi,
> Is it possible to configure slurm/cgroups in such way that jobs that are
> using more memory than they asked for are not killed if there still are
> free memory available on the compute node? When free memory gets low
> these jobs can be killed as usual.
> 
> Today when a job has exceeded its limits it is killed immediately. Since
> the applications only requires maximum memory for a short period of time
> we can often not run as many concurrent jobs as we want.
> 
> Maybe I can rephrase the question a bit: How can you configure memory
> limits for a job when the job only needs maximum memory during a short
> time? Example: Job1 needs 80G RAM but only during 15% of the execution
> time, during the remaining 85% it only needs 30G.
> 
> I guess the obvious thing is to use "CR_Core" instead of
> "CR_Core_Memory" we use today. But we have to constrain memory in some
> way because the nodes are also running daemons for the distributed file
> system and that must not be affected by running jobs.
> 
> Any ideas?
> 
> Regards,
> Alexander
> 
> 

It's possible by setting the cgroup soft and hard limits appropriately.
See https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt for
details.

Unfortunately the way the slurm cgroup.conf configuration maps to the
actual cgroups is not exactly intuitive (one can hope that when
eventually slurm starts to support cgroupsv2 this would be fixed...).

For example, if you want to set the cgroup soft limit to the amount of
memory that the job requests with --mem= or --mem-per-core= options, and
then set the hard limit (when the job will be killed) to 5x the soft limit:

ConstrainRAMSpace=no
ConstrainSwapSpace=yes
AllowedSwapSpace=400

(Note that it's not possible to separately set the maximum swap usage.
If you instead limit only the memory and not mem+swap, it will limit
memory but swap usage will be unlimited.)

As for you second part of the question, no, it's not possible to have
varying limits during a job.

-- 
Janne Blomqvist, D.Sc. (Tech.), Scientific Computing Specialist
Aalto University School of Science, PHYS & NBE
+358503841576 || janne.blomqvist at aalto.fi