<div dir="ltr"><div>Hi Sean,</div><div><br></div><div>I don't see PrologFlags=Contain in your slurm.conf. It is one of the entries required to activate the cgroup containment: <a href="https://slurm.schedmd.com/cgroup.conf.html#OPT_/etc/slurm/slurm.conf">https://slurm.schedmd.com/cgroup.conf.html#OPT_/etc/slurm/slurm.conf</a></div><div><br></div><div>Best,</div><div><br></div><div>-Sean<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 8, 2022 at 8:16 AM Sean McGrath <<a href="mailto:smcgrat@tchpc.tcd.ie">smcgrat@tchpc.tcd.ie</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I can't get cgroups to constrain memory or cores. If anyone is able to point out what I am doing wrong I would be very grateful please. <br>
<br>
Testing:<br>
<br>
Request a core and 2G of memory, log into it and compile a binary that just allocates memory quickly:<br>
<br>
$ salloc -n 1 --mem=2G<br>
$ ssh $SLURM_NODELIST<br>
$ cat stoopid-memory-overallocation.c <br>
/*<br>
* <br>
* Sometimes you need to over allocate the memory available to you.<br>
* This does so splendidly. I just hope you have limits set to kill it!<br>
* <br>
*/<br>
<br>
int main()<br>
{<br>
while(1)<br>
{<br>
void *m = malloc(1024*1024);<br>
memset(m,0,1024*1024);<br>
}<br>
return 0;<br>
}<br>
$ gcc -o stoopid-memory-overallocation.x stoopid-memory-overallocation.c<br>
<br>
Checking memory usage before as a baseline:<br>
<br>
$ free -g<br>
total used free shared buff/cache available<br>
Mem: 251 1 246 0 3 248<br>
Swap: 7 0 7<br>
<br>
Launch the memory over allocation and check memory use subsequently and see that 34G has been allocated when I expect it to be constrained to 2G:<br>
<br>
$ ./stoopid-memory-overallocation.x &<br>
$ sleep 10 && free -g<br>
total used free shared buff/cache available<br>
Mem: 251 34 213 0 3 215<br>
Swap: 7 0 7<br>
<br>
Run another process to check cpu constraints:<br>
<br>
$ ./stoopid-memory-overallocation.x &<br>
<br>
Check it with top and I can see that the 2 processes are running simultaneously:<br>
<br>
$ top<br>
top - 13:04:44 up 13 days, 23:39, 2 users, load average: 0.63, 0.27, 0.11<br>
Tasks: 525 total, 3 running, 522 sleeping, 0 stopped, 0 zombie<br>
%Cpu(s): 0.7 us, 5.5 sy, 0.0 ni, 93.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>
MiB Mem : 257404.1 total, 181300.3 free, 72588.6 used, 3515.2 buff/cache<br>
MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 183300.3 avail Mem <br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
120978 smcgrat 20 0 57.6g 57.6g 968 R 100.0 22.9 0:22.63 stoopid-memory-<br>
120981 smcgrat 20 0 11.6g 11.6g 952 R 100.0 4.6 0:04.57 stoopid-memory-<br>
...<br>
<br>
Is this actually a valid test case or am I doing something else wrong?<br>
<br>
Thanks<br>
<br>
Sean<br>
<br>
Setup details:<br>
<br>
Ubuntu 20.04.5 LTS (Focal Fossa). <br>
slurm 21.08.8-2. <br>
cgroup-tools version 0.41-10 installed. <br>
<br>
The following was set in /etc/default/grub and update-grub run:<br>
<br>
GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"<br>
<br>
Relevant parts of scontrol show conf<br>
<br>
JobAcctGatherType = jobacct_gather/none<br>
ProctrackType = proctrack/cgroup<br>
TaskPlugin = task/cgroup<br>
TaskPluginParam = (null type)<br>
<br>
<br>
The contents of the full slurm.conf<br>
<br>
ClusterName=neuro<br>
SlurmctldHost=neuro01(192.168.49.254)<br>
AuthType=auth/munge<br>
CommunicationParameters=block_null_hash<br>
CryptoType=crypto/munge<br>
Epilog=/home/support/slurm/etc/slurm.epilog.local<br>
EpilogSlurmctld=/home/support/slurm/etc/slurm.epilogslurmctld<br>
JobRequeue=0<br>
MaxJobCount=30000<br>
MpiDefault=none<br>
Prolog=/home/support/slurm/etc/prolog<br>
ReturnToService=2<br>
SlurmctldPidFile=/var/run/slurmctld.pid<br>
SlurmctldPort=6817<br>
SlurmdPidFile=/var/run/slurmd.pid<br>
SlurmdPort=6818<br>
SlurmUser=root<br>
StateSaveLocation=/var/slurm_state/neuro<br>
SwitchType=switch/none<br>
TaskPlugin=task/cgroup<br>
ProctrackType=proctrack/cgroup<br>
RebootProgram=/sbin/reboot<br>
InactiveLimit=0<br>
KillWait=30<br>
MinJobAge=300<br>
SlurmctldTimeout=300<br>
SlurmdTimeout=300<br>
Waittime=0<br>
SchedulerType=sched/backfill<br>
SelectType=select/cons_res<br>
SelectTypeParameters=CR_Core<br>
AccountingStorageHost=service01<br>
AccountingStorageType=accounting_storage/slurmdbd<br>
JobCompType=jobcomp/none<br>
JobAcctGatherFrequency=30<br>
SlurmctldDebug=3<br>
SlurmctldLogFile=/var/log/slurm.log<br>
SlurmdDebug=3<br>
SlurmdLogFile=/var/log/slurm.log<br>
DefMemPerNode=257300<br>
MaxMemPerNode=257300<br>
NodeName=neuro-n01-mgt RealMemory=257300 Sockets=2 CoresPerSocket=16 State=UNKNOWN<br>
NodeName=neuro-n02-mgt RealMemory=257300 Sockets=2 CoresPerSocket=16 State=UNKNOWN<br>
PartitionName=compute Nodes=ALL Default=YES MaxTime=5760 State=UP Shared=YES<br>
<br>
<br>
cgroup.conf file contents:<br>
<br>
CgroupAutomount=yes<br>
ConstrainCores=yes<br>
ConstrainRAMSpace=yes<br>
TaskAffinity=no<br>
<br>
</blockquote></div>