[slurm-users] slurm, memory accounting and memory mapping

[Sergey Koposov]

Hi, 

I've recently migrated to slurm from pbs on our cluster. Because of that, now the job memory limits are
strictly enforced and that causes my code to get killed. 
The trick is that my code uses memory mapping (i.e. mmap) of one single large file (~12 Gb) in each thread on each node.
With this technique in the past despite the fact the file is (read-only) mmaped in say 16 threads, the actual memory footprint was still ~ 12 Gb.
However, when I now do this in slurm, it thinks that each thread (or process) takes 12Gb and kills my processes. 

Does anyone has a way around this problem ? Other then stoping using Memory as a consumable resource, or faking that each node has more memory ? 

Here is an example slurm script that I'm running 
#!/bin/bash
#SBATCH -N 1 # number of nodes
#SBATCH --cpus-per-task=10 # number of cores
#SBATCH --ntasks-per-node=1
#SBATCH --mem=125GB
#SBATCH --array=0-4

sh script1.sh $SLURM_ARRAY_TASK_ID 5 

The script1 essentially starts python which in turn create 10 multiprocessing processes each of which will mmap the large file. 
------
In this case I'm forced to limit myself to using only 10 threads, instead of 16 (our machines have 16 cores) to avoid being killed by slurm. 
---
Thanks in advance for any suggestions.
         
            Sergey