[slurm-users] Tracking maximum memory via cgroup

[Patrik Andersson]

Looking into this more it looks like memory.max_usage_in_byte and
memory.usage_in_bytes also count file cache. Which is very surprising and
not at all useful. But total_rss in memory.stat shows a more correct
number. Looking at that one for a real job gives me around 30 GB, which
matches my other data and expectations. But my issue remains that sstat and
sacct report only a few MB for memory stats.

On Mon, 9 Nov 2020 at 18:49, Patrik Andersson <patrik.s.andersson at gmail.com>
wrote:

> We are using cgroups to track resource usage of our jobs. The jobs are run
> in docker with docker's --parent-cgroup flag pointing at the slurm job's
> cgroup. This works great for limiting memory usage.
>
> Unfortunately the maximum memory usage, maxRSS, is not accurately reported
> in sacct. While the cgroup's memory.max_usage_in_bytes does show accurate
> numbers.
>
> Looking at the cgroup:
>
>> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.max_usage_in_bytes:1132154880
>> # 1GB
>> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.use_hierarchy:1
>> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:rss 0
>> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:total_rss 524288
>>
>
> Looking at sacct:
>
>> $ sacct -j 31626 -o
>> jobid,AveRSS,MaxRSS,AveVMSize,MaxVMSize,ReqMem,TotalCPU
>>
>>        JobID     AveRSS     MaxRSS  MaxVMSize
>> 31626.batch      28600K     28600K     77900K
>
>
> I expected that we would get some of the cgroup stats since we are using
> cgroup plugins.
>
> lines from slurm.conf
>
>> JobAcctGatherFrequency=30
>>
>> JobAcctGatherType=jobacct_gather/cgroup
>>
>> ProctrackType=proctrack/cgroup
>>
>> TaskPlugin=task/affinity,task/cgroup
>>
>> SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK
>>
>
>  cgroup.conf
>
>> CgroupAutomount=yes
>>
>> CgroupMountpoint=/sys/fs/cgroup
>>
>>
>>
>> ### Task/cgroup Plugin ###
>>
>> # Constrain allowed cores to the subset of allocated resources.
>>
>> # This functionality makes use of the cpuset subsystem
>>
>> ConstrainCores=yes
>>
>> ConstrainKmemSpace=yes
>>
>> ConstrainRAMSpace=yes
>>
>> ConstrainSwapSpace=yes
>>
>> ConstrainDevices=no
>>
>> MinKmemSpace=30
>>
>> MinRAMSpace=30
>>
>> # Set a default task affinity to bind each step task to a subset of the
>>
>> # allocated cores using sched_setaffinity
>>
>> # /!\ This feature requires the Portable Hardware Locality (hwloc) library
>>
>> TaskAffinity=no
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201110/449c3802/attachment.htm>