[slurm-users] Tracking maximum memory via cgroup

[Patrik Andersson]

We are using cgroups to track resource usage of our jobs. The jobs are run
in docker with docker's --parent-cgroup flag pointing at the slurm job's
cgroup. This works great for limiting memory usage.

Unfortunately the maximum memory usage, maxRSS, is not accurately reported
in sacct. While the cgroup's memory.max_usage_in_bytes does show accurate
numbers.

Looking at the cgroup:

> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.max_usage_in_bytes:1132154880
> # 1GB
> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.use_hierarchy:1
> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:rss 0
> /sys/fs/cgroup/memory/slurm/uid_500/job_31626/memory.stat:total_rss 524288
>

Looking at sacct:

> $ sacct -j 31626 -o jobid,AveRSS,MaxRSS,AveVMSize,MaxVMSize,ReqMem,TotalCPU
>
>        JobID     AveRSS     MaxRSS  MaxVMSize
> 31626.batch      28600K     28600K     77900K


I expected that we would get some of the cgroup stats since we are using
cgroup plugins.

lines from slurm.conf

> JobAcctGatherFrequency=30
>
> JobAcctGatherType=jobacct_gather/cgroup
>
> ProctrackType=proctrack/cgroup
>
> TaskPlugin=task/affinity,task/cgroup
>
> SelectTypeParameters=CR_Core_Memory,CR_CORE_DEFAULT_DIST_BLOCK
>

 cgroup.conf

> CgroupAutomount=yes
>
> CgroupMountpoint=/sys/fs/cgroup
>
>
>
> ### Task/cgroup Plugin ###
>
> # Constrain allowed cores to the subset of allocated resources.
>
> # This functionality makes use of the cpuset subsystem
>
> ConstrainCores=yes
>
> ConstrainKmemSpace=yes
>
> ConstrainRAMSpace=yes
>
> ConstrainSwapSpace=yes
>
> ConstrainDevices=no
>
> MinKmemSpace=30
>
> MinRAMSpace=30
>
> # Set a default task affinity to bind each step task to a subset of the
>
> # allocated cores using sched_setaffinity
>
> # /!\ This feature requires the Portable Hardware Locality (hwloc) library
>
> TaskAffinity=no
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20201109/a71034bd/attachment.htm>