[slurm-users] pam_slurm_adopt does not constrain memory?

[Andy Georges]

Hi,

> On 22 Aug 2018, at 16:27, Christian Peter <christian.peter at itwm.fraunhofer.de> wrote:
> 
> hi,
> 
> we observed a strange behavior of pam_slurm_adopt regarding the involved cgroups:
> 
> when we start a shell as a new Slurm job using "srun", the process has freezer, cpuset and memory cgroups setup as e.g. "/slurm/uid_5001/job_410318/step_0". that's good!
> 
> however, another shell started by an SSH login is handled by pam_slurm_adopt. that process is only affected by the freezer and cpuset cgroups setup as "/slurm/uid_5001/job_410318/step_extern". it lacks the configuration of the "memory" cgroup. (see output below)
> 
> as a consequence, all tools started from this shell prompt are not affected by any memory restrictions. that's bad for our use case as we need to partition the memory of our SMP machines for several independent jobs/users.
> 
> is this an expected behavior of pam_slurm_adopt/slurmstepd?
> or maybe a configuration issue? did i miss something?
> 
> a bug? to me, it looks similar to this old issue...
> https://bugs.schedmd.com/show_bug.cgi?id=2236
> 
> we're currently running Slurm 17.11.8. (we've already seen this with our previous version 17.11.5.)


See https://github.com/hpcugent/slurm/pull/28 and https://bugs.schedmd.com/show_bug.cgi?id=5920 for a potential workaround (and discussion, I hope) that does not require you to disable systemd-logind. 


We now get:

vsc40075 at node2801 (banette) ~> cat /proc/self/cgroup
11:hugetlb:/
10:pids:/
9:freezer:/slurm/uid_2540075/job_67119295/step_extern
8:perf_event:/
7:cpuset:/slurm/uid_2540075/job_67119295/step_extern
6:devices:/
5:net_prio,net_cls:/
4:memory:/slurm/uid_2540075/job_67119295/step_extern/task_0
3:blkio:/
2:cpuacct,cpu:/slurm/uid_2540075/job_67119295/step_extern/task_0
1:name=systemd:/user.slice/user-2540075.slice/session-9963.scope


when logging into a compute node that has a job running, using the following PAM sushi config lines:

account sufficient pam_slurm_adopt.so action_no_jobs=deny,action_adopt_failure=deny,debug,action_adopt=check_only
session sufficient pam_slurm_adopt.so action_no_jobs=ignore,action_adopt_failure=deny,debug


Kind regards,
— Andy