[slurm-users] ulimit in sbatch script

[Mahmood Naderan]

Hi Bill,
Sorry for the late reply. As I greped for pam_limits.so, I see

[root at rocks7 ~]# grep -r pam_limits.so /etc/pam.d/
/etc/pam.d/sudo:session    required     pam_limits.so
/etc/pam.d/runuser:session              required        pam_limits.so
/etc/pam.d/sudo-i:session    required     pam_limits.so
/etc/pam.d/system-auth-ac:session     required      pam_limits.so
/etc/pam.d/fingerprint-auth-ac:session     required      pam_limits.so
/etc/pam.d/smartcard-auth-ac:session     required      pam_limits.so
/etc/pam.d/password-auth-ac:session     required      pam_limits.so
[root at rocks7 ~]# grep -r UsePAM /etc/slurm/
/etc/slurm/slurm.conf:#UsePAM=



So, UsePAM has not been set. So, slurm shouldn't limit anything. Is
that correct? however, I see that slurm limits the virtual memory size


[hamid at rocks7 case1_source2]$ cat slurm_script.sh
#!/bin/bash
#SBATCH --job-name=hvacSteadyFoam
#SBATCH --output=hvacSteadyFoam.log
#SBATCH --ntasks=32
#SBATCH --time=100:00:00
#SBATCH --mem=64000M
ulimit -a
mpirun hvacSteadyFoam -parallel

[hamid at rocks7 case1_source2]$ sbatch slurm_script.sh
Submitted batch job 55
[hamid at rocks7 case1_source2]$ ssh compute-0-3
Warning: untrusted X11 forwarding setup failed: xauth key data not generated
Last login: Sun Apr 15 23:11:15 2018 from rocks7.local
Rocks Compute Node
Rocks 7.0 (Manzanita)
Profile built 19:21 11-Apr-2018

Kickstarted 19:37 11-Apr-2018
[hamid at compute-0-3 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256712
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[hamid at compute-0-3 ~]$ exit
logout
Connection to compute-0-3 closed.
[hamid at rocks7 case1_source2]$ cat hvacSteadyFoam.log
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256712
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 65536000
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) 72089600
file locks                      (-x) unlimited
[hamid at rocks7 case1_source2]$


Regards,
Mahmood




On Mon, Apr 16, 2018 at 12:02 AM, Bill Barth <bbarth at tacc.utexas.edu> wrote:
> Specifying --mem to Slurm only tells it to find a node that has that much, not to enforce a limit as far as I know. That node has that much so it finds it. You probably want to enable UsePAM and setup the pam.d slurm files and /etc/security/limits.conf to keep users under the 64000MB physical memory that the node has (minus some padding for the OS, etc.). IS UsePAM enabled in your slurm.conf, maybe that’s doing it.
>
> Best,
> Bill.
>
> --
> Bill Barth, Ph.D., Director, HPC
> bbarth at tacc.utexas.edu        |   Phone: (512) 232-7069
> Office: ROC 1.435            |   Fax:   (512) 475-9445
>