[slurm-users] Virtual memory size requested by slurm

[Wensheng Deng]

Hi Mahmood,
You may check here and see if it helps you:
https://unix.stackexchange.com/questions/345595/how-to-set-ulimits-on-service-with-systemd



On Mon, Jan 27, 2020 at 2:41 PM Mahmood Naderan <mahmood.nt at gmail.com>
wrote:

> Excuse me, I see that "ulimit -a" shows unlimited virtual memory size when
> the user runs that on terminal. However, when he puts the command in sbatch
> script, the value is limited to about 40GB.
>
> [shams at hpc ~]$ ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 256657
> max locked memory       (kbytes, -l) unlimited
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 4096
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> [shams at hpc ~]$ cat slurm_blast.sh
> #!/bin/bash
> #SBATCH --job-name=blast1
> #SBATCH --output=my_blast.log
> #SBATCH --partition=SEA
> #SBATCH --account=fish
> #SBATCH --mem=38GB
> #SBATCH --nodelist=hpc
> #SBATCH --nodes=1
> #SBATCH --ntasks-per-node=2
> export PATH=~/ncbi-blast-2.9.0+/bin:$PATH
> ulimit -a | grep virtual
> ulimit -v 146800640
> ulimit -a | grep virtual
> blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta
> -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2
> [shams at hpc ~]$ sbatch slurm_blast.sh
> Submitted batch job 304
> [shams at hpc ~]$ cat my_blast.log
> virtual memory          (kbytes, -v) 43830476
> /var/spool/slurmd/job00304/slurm_script: line 13: ulimit: virtual memory:
> cannot modify limit: Operation not permitted
> virtual memory          (kbytes, -v) 43830476
> Error memory mapping:/home/shams/ncbi-blast-2.9.0+/bin/nr.52.psq
> openedFilesCount=151 threadID=0
> Error: NCBI C++ Exception:
>     T0
> "/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_560232_130.14.18.6_9008__PrepareRelease_Linux64-Centos_1552331742/c++/compilers/unix/../../src/corelib/ncbiobj.cpp",
> line 981: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to
> access NULL pointer.
>      Stack trace:
>       blastx ???:0 ncbi::CStackTraceImpl::CStackTraceImpl() offset=0x77
> addr=0x1d95da7
>       blastx ???:0 ncbi::CStackTrace::CStackTrace(std::string const&)
> offset=0x25 addr=0x1d98465
>       blastx ???:0 ncbi::CException::x_GetStackTrace() offset=0xA0
> addr=0x1ec7330
>       blastx ???:0 ncbi::CException::SetSeverity(ncbi::EDiagSev)
> offset=0x49 addr=0x1ec2169
>       blastx ???:0 ncbi::CObject::ThrowNullPointerException() offset=0x2D2
> addr=0x1f42582
>       blastx ???:0 ncbi::blast::CBlastTracebackSearch::Run() offset=0x61C
> addr=0xf2929c
>       blastx ???:0 ncbi::blast::CLocalBlast::Run() offset=0x404
> addr=0xed4684
>       blastx ???:0 CBlastxApp::Run() offset=0xC9C addr=0x9cbf7c
>       blastx ???:0 ncbi::CNcbiApplication::x_TryMain(ncbi::EAppDiagStream,
> char const*, int*, bool*) offset=0x8E3 addr=0x1da0e13
>       blastx ???:0 ncbi::CNcbiApplication::AppMain(int, char const*
> const*, char const* const*, ncbi::EAppDiagStream, char const*, std::string
> const&) offset=0x782 addr=0x1d9f6b2
>       blastx ???:0 main offset=0x5E5 addr=0x9caa05
>       /lib64/libc.so.6 ???:0 __libc_start_main offset=0xF5
> addr=0x7ff1d519e505
>       blastx ???:0 blastx() [0x9ca345] offset=0x0 addr=0x9ca345
>
>
>
> Moreover, the user can not change the virtual size.
> I really wonder how can I  tell slurm not to change that parameter!!
>
>
> Regards,
> Mahmood
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200127/5984f7f5/attachment-0001.htm>