[slurm-users] MaxMemPerCPU not enforced?

[Angel de Vicente]

Hello,

I'm trying to get Slurm to control the memory used per CPU, but it does
not seem to enforce the MaxMemPerCPU option in slurm.conf

This is running in Ubuntu 22.04 (cgroups v2), Slurm 23.02.3.

Relevant configuration options:

,----cgroup.conf
| AllowedRAMSpace=100
| ConstrainCores=yes
| ConstrainRAMSpace=yes
| ConstrainSwapSpace=yes
| AllowedSwapSpace=0
`----

,----slurm.conf
| TaskPlugin=task/affinity,task/cgroup
| PrologFlags=X11
| 
| SelectType=select/cons_res
| SelectTypeParameters=CR_CPU_Memory,CR_CORE_DEFAULT_DIST_BLOCK
| MaxMemPerCPU=500
| DefMemPerCPU=200
| 
| JobAcctGatherType=jobacct_gather/linux
| 
| EnforcePartLimits=ALL
| 
| NodeName=xxx RealMemory=257756 Sockets=4 CoresPerSocket=8 ThreadsPerCore=1 Weight=1
| 
| PartitionName=batch       Nodes=duna State=UP Default=YES MaxTime=2-00:00:00 MaxCPUsPerNode=32 OverSubscribe=FORCE:1
| PartitionName=interactive Nodes=duna State=UP Default=NO  MaxTime=08:00:00   MaxCPUsPerNode=32 OverSubscribe=FORCE:2
`----


I can ask for an interactive session with 4GB/CPU (I would have thought
that "EnforcePartLimits=ALL" would stop me from doing that), and once
I'm in the interactive session I can execute a 3GB test code without any
issues (I can see with htop that the process does indeed use a RES size
of 3GB at 100% CPU use). Any idea what could be the problem or how to
start debugging this?

,----
| [angelv at xxx test]$ sinter -n 1 --mem-per-cpu=4000
| salloc: Granted job allocation 127544
| salloc: Nodes xxx are ready for job
| 
| (sinter) [angelv at xxx test]$ stress -m 1 -t 600 --vm-keep --vm-bytes 3G
| stress -m 1 -t 600 --vm-keep --vm-bytes 3G
| stress: info: [1772392] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
`----

Many thanks,
-- 
Ángel de Vicente
 Research Software Engineer (Supercomputing and BigData)
 Tel.: +34 922-605-747
 Web.: http://research.iac.es/proyecto/polmag/

 GPG: 0x8BDC390B69033F52
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5877 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230724/035c4df8/attachment.bin>