<!DOCTYPE html><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /></head><body><div data-html-editor-font-wrapper="true" style="font-family: arial, sans-serif; font-size: 13px;">Hi,<br><br>Many thanks for that pointer Sean. I had missed the PrologFlags=Contain setting so have added it to slurm.conf now.<br><br>I've also explicitly built slurm with pam support:<br><br>../configure --sysconfdir=/home/support/pkgs/slurm/etc --prefix=/home/support/pkgs/slurm/ubuntu_20.04/21.08.8-2 --localstatedir=/var/run/slurm --enable-pam && make && make install install<br><br>It appears to me as if slurm tasks are launching within cgroups.<br><br>E.g. if I do<br><br>srun --mem=100 sleep 300&<br><br>And login to the node I can see memory limits for cgroups:<br><br>$ cat /sys/fs/cgroup/memory/slurm/uid_5446/memory.limit_in_bytes<br>9223372036854771712<br>$ cat /sys/fs/cgroup/memory/slurm/uid_5446/job_24/memory.limit_in_bytes<br>269907656704<br>$ cat /sys/fs/cgroup/memory/slurm/uid_5446/job_24/step_0/memory.limit_in_bytes <br>269907656704<br><br>But if I do this to over allocate memory it still allows me to:<br><br>srun --mem=100 stoopid-memory-overallocation.x<br><br>More memory is being allocated by the node than should be allowed.<br><br>I'm clearly doing something wrong here. Can anyone point out what it is please? Am I just using the wrong test methodology?<br><br>Thanks in advance <br><br>Sean<br><br>November 8, 2022 1:48 PM, "Sean Maxwell" <<a target="_blank" tabindex="-1" href="mailto:stm@case.edu?to=%22Sean%20Maxwell%22%20<stm@case.edu>">stm@case.edu</a>> wrote:<br> <blockquote><div><div><div dir="ltr"><div>Hi Sean,</div><div></div><div>I don't see PrologFlags=Contain in your slurm.conf. It is one of the entries required to activate the cgroup containment: <a rel="external nofollow noopener noreferrer" target="_blank" tabindex="-1" href="https://slurm.schedmd.com/cgroup.conf.html#OPT_/etc/slurm/slurm.conf">https://slurm.schedmd.com/cgroup.conf.html#OPT_/etc/slurm/slurm.conf</a></div><div></div><div>Best,</div><div></div><div>-Sean</div></div> <div><div dir="ltr">On Tue, Nov 8, 2022 at 8:16 AM Sean McGrath <<a rel="external nofollow noopener noreferrer" target="_blank" tabindex="-1" href="mailto:smcgrat@tchpc.tcd.ie">smcgrat@tchpc.tcd.ie</a>> wrote:</div> <blockquote style="margin: 0px 0px 0px 0.8ex;border-left: 1px solid rgb(204,204,204);padding-left: 1ex">Hi,<br><br>I can't get cgroups to constrain memory or cores. If anyone is able to point out what I am doing wrong I would be very grateful please.<br><br>Testing:<br><br>Request a core and 2G of memory, log into it and compile a binary that just allocates memory quickly:<br><br>$ salloc -n 1 --mem=2G<br>$ ssh $SLURM_NODELIST<br>$ cat stoopid-memory-overallocation.c<br>/*<br>*<br>* Sometimes you need to over allocate the memory available to you.<br>* This does so splendidly. I just hope you have limits set to kill it!<br>*<br>*/<br><br>int main()<br>{<br>while(1)<br>{<br>void *m = malloc(1024*1024);<br>memset(m,0,1024*1024);<br>}<br>return 0;<br>}<br>$ gcc -o stoopid-memory-overallocation.x stoopid-memory-overallocation.c<br><br>Checking memory usage before as a baseline:<br><br>$ free -g<br>total used free shared buff/cache available<br>Mem: 251 1 246 0 3 248<br>Swap: 7 0 7<br><br>Launch the memory over allocation and check memory use subsequently and see that 34G has been allocated when I expect it to be constrained to 2G:<br><br>$ ./stoopid-memory-overallocation.x &<br>$ sleep 10 && free -g<br>total used free shared buff/cache available<br>Mem: 251 34 213 0 3 215<br>Swap: 7 0 7<br><br>Run another process to check cpu constraints:<br><br>$ ./stoopid-memory-overallocation.x &<br><br>Check it with top and I can see that the 2 processes are running simultaneously:<br><br>$ top<br>top - 13:04:44 up 13 days, 23:39, 2 users, load average: 0.63, 0.27, 0.11<br>Tasks: 525 total, 3 running, 522 sleeping, 0 stopped, 0 zombie<br>%Cpu(s): 0.7 us, 5.5 sy, 0.0 ni, 93.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br>MiB Mem : 257404.1 total, 181300.3 free, 72588.6 used, 3515.2 buff/cache<br>MiB Swap: 8192.0 total, 8192.0 free, 0.0 used. 183300.3 avail Mem<br><br>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>120978 smcgrat 20 0 57.6g 57.6g 968 R 100.0 22.9 0:22.63 stoopid-memory-<br>120981 smcgrat 20 0 11.6g 11.6g 952 R 100.0 4.6 0:04.57 stoopid-memory-<br>...<br><br>Is this actually a valid test case or am I doing something else wrong?<br><br>Thanks<br><br>Sean<br><br>Setup details:<br><br>Ubuntu 20.04.5 LTS (Focal Fossa).<br>slurm 21.08.8-2.<br>cgroup-tools version 0.41-10 installed.<br><br>The following was set in /etc/default/grub and update-grub run:<br><br>GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"<br><br>Relevant parts of scontrol show conf<br><br>JobAcctGatherType = jobacct_gather/none<br>ProctrackType = proctrack/cgroup<br>TaskPlugin = task/cgroup<br>TaskPluginParam = (null type)<br><br><br>The contents of the full slurm.conf<br><br>ClusterName=neuro<br>SlurmctldHost=neuro01(192.168.49.254)<br>AuthType=auth/munge<br>CommunicationParameters=block_null_hash<br>CryptoType=crypto/munge<br>Epilog=/home/support/slurm/etc/slurm.epilog.local<br>EpilogSlurmctld=/home/support/slurm/etc/slurm.epilogslurmctld<br>JobRequeue=0<br>MaxJobCount=30000<br>MpiDefault=none<br>Prolog=/home/support/slurm/etc/prolog<br>ReturnToService=2<br>SlurmctldPidFile=/var/run/slurmctld.pid<br>SlurmctldPort=6817<br>SlurmdPidFile=/var/run/slurmd.pid<br>SlurmdPort=6818<br>SlurmUser=root<br>StateSaveLocation=/var/slurm_state/neuro<br>SwitchType=switch/none<br>TaskPlugin=task/cgroup<br>ProctrackType=proctrack/cgroup<br>RebootProgram=/sbin/reboot<br>InactiveLimit=0<br>KillWait=30<br>MinJobAge=300<br>SlurmctldTimeout=300<br>SlurmdTimeout=300<br>Waittime=0<br>SchedulerType=sched/backfill<br>SelectType=select/cons_res<br>SelectTypeParameters=CR_Core<br>AccountingStorageHost=service01<br>AccountingStorageType=accounting_storage/slurmdbd<br>JobCompType=jobcomp/none<br>JobAcctGatherFrequency=30<br>SlurmctldDebug=3<br>SlurmctldLogFile=/var/log/slurm.log<br>SlurmdDebug=3<br>SlurmdLogFile=/var/log/slurm.log<br>DefMemPerNode=257300<br>MaxMemPerNode=257300<br>NodeName=neuro-n01-mgt RealMemory=257300 Sockets=2 CoresPerSocket=16 State=UNKNOWN<br>NodeName=neuro-n02-mgt RealMemory=257300 Sockets=2 CoresPerSocket=16 State=UNKNOWN<br>PartitionName=compute Nodes=ALL Default=YES MaxTime=5760 State=UP Shared=YES<br><br><br>cgroup.conf file contents:<br><br>CgroupAutomount=yes<br>ConstrainCores=yes<br>ConstrainRAMSpace=yes<br>TaskAffinity=no<br> </blockquote></div></div></div></blockquote><br><br><signature></signature></div></body></html>