[slurm-users] Gentle memory limits in Slurm using cgroup?

[Alexander Åhman]

Thanks for the info!
Didn't know Slurm supported to set a soft limit. Did have a look in the 
code and found function "memcg_initialize" in file 
"plugins/task/cgroup/task_cgroup_memory.c".

It looks like in order to set both hard and soft limits you need to do:

ConstrainRAMSpace=yes
AllowedRAMSpace=200

This should allow jobs to use 200% the requested memory. So if a job 
request --mem=10G then it will be able to use 10G RAM but will not be 
killed unless it uses 20G (if there are free RAM available), but it will 
also consume all swap first.

After some testing, this works just fine if I have ConstrainSwapSpace=no 
but if I change that to yes I get strange behaviours. At least the 
output from sacct is not what I expect it to be. I'm running the exact 
same test program (just trying to get as much memory as possible) but in 
the second run I have ConstrainSwapSpace=yes. Why is it not recording 
the correct(?) amount of memory used?

ConstrainRAMSpace=yes
AllowedRAMSpace=150

         JobID      State ExitCode     MaxRSS 
MaxVMSize                        AllocTRES
------------- ---------- -------- ---------- ---------- 
--------------------------------
        911124 OUT_OF_ME+    0:125 billing=1,cpu=1,mem=2G,node=1
911124.extern  COMPLETED      0:0      1104K    171896K 
billing=1,cpu=1,mem=2G,node=1
      911124.0 OUT_OF_ME+    0:125   3141964K 14340268K              
cpu=1,mem=2G,node=1


ConstrainRAMSpace=yes
AllowedRAMSpace=150
ConstrainSwapSpace=yes
AllowedSwapSpace=200

         JobID      State ExitCode     MaxRSS 
MaxVMSize                        AllocTRES
------------- ---------- -------- ---------- ---------- 
--------------------------------
        911081 OUT_OF_ME+    0:125 billing=1,cpu=1,mem=2G,node=1
911081.extern  COMPLETED      0:0      1096K    171896K 
billing=1,cpu=1,mem=2G,node=1
      911081.0 OUT_OF_ME+    0:125      1216K    240648K          
cpu=1,mem=2G,node=1


Memory is complicated stuff...

Regards,
Alexander


On 03/05/2019 08:20, Janne Blomqvist wrote:
> On 02/05/2019 17.53, Alexander Åhman wrote:
>> Hi,
>> Is it possible to configure slurm/cgroups in such way that jobs that are
>> using more memory than they asked for are not killed if there still are
>> free memory available on the compute node? When free memory gets low
>> these jobs can be killed as usual.
>>
>> Today when a job has exceeded its limits it is killed immediately. Since
>> the applications only requires maximum memory for a short period of time
>> we can often not run as many concurrent jobs as we want.
>>
>> Maybe I can rephrase the question a bit: How can you configure memory
>> limits for a job when the job only needs maximum memory during a short
>> time? Example: Job1 needs 80G RAM but only during 15% of the execution
>> time, during the remaining 85% it only needs 30G.
>>
>> I guess the obvious thing is to use "CR_Core" instead of
>> "CR_Core_Memory" we use today. But we have to constrain memory in some
>> way because the nodes are also running daemons for the distributed file
>> system and that must not be affected by running jobs.
>>
>> Any ideas?
>>
>> Regards,
>> Alexander
>>
>>
> It's possible by setting the cgroup soft and hard limits appropriately.
> See https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt for
> details.
>
> Unfortunately the way the slurm cgroup.conf configuration maps to the
> actual cgroups is not exactly intuitive (one can hope that when
> eventually slurm starts to support cgroupsv2 this would be fixed...).
>
> For example, if you want to set the cgroup soft limit to the amount of
> memory that the job requests with --mem= or --mem-per-core= options, and
> then set the hard limit (when the job will be killed) to 5x the soft limit:
>
> ConstrainRAMSpace=no
> ConstrainSwapSpace=yes
> AllowedSwapSpace=400
>
> (Note that it's not possible to separately set the maximum swap usage.
> If you instead limit only the memory and not mem+swap, it will limit
> memory but swap usage will be unlimited.)
>
> As for you second part of the question, no, it's not possible to have
> varying limits during a job.
>