[slurm-users] blastx fails with "Error memory mapping"

[Jeffrey T Frey]

Does your Slurm cgroup or node OS cgroup configuration limit the virtual address space of processes?  The "Error memory mapping" is thrown by blast when trying to create a virtual address space that exposes the contents of a file on disk (see "man mmap") so the file can be accessed via pointers (with the OS handling paging data in and out of the file on disk) rather than by means of standard file i/o calls (e.g. fread(), fscanf(), read()).  It sounds like you don't have enough system RAM, period, or the cgroup "memory.memsw.limit_in_bytes" is set too low for the amount of file content you're attempting to mmap() into the virtual address space (e.g. BIG files).




> On Jan 24, 2020, at 07:03 , Mahmood Naderan <mahmood.nt at gmail.com> wrote:
> 
> Hi,
> Although I can run the blastx command on terminal on all nodes, I can not use slurm for that due to a so called "memory map error".
> Please see below that I pressed ^C after some seconds when running via terminal.
> 
> Fri Jan 24 15:29:57 +0330 2020
> [shams at hpc ~]$ blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2
> ^C
> [shams at hpc ~]$ date
> Fri Jan 24 15:30:09 +0330 2020
> 
> 
> However, the following script fails
> 
> [shams at hpc ~]$ cat slurm_blast.sh
> #!/bin/bash
> #SBATCH --job-name=blast1
> #SBATCH --output=my_blast.log
> #SBATCH --partition=SEA
> #SBATCH --account=fish
> #SBATCH --mem=38GB
> #SBATCH --nodelist=hpc
> #SBATCH --nodes=1
> #SBATCH --ntasks-per-node=2
> 
> export PATH=~/ncbi-blast-2.9.0+/bin:$PATH
> blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2
> [shams at hpc ~]$ sbatch slurm_blast.sh
> Submitted batch job 284
> [shams at hpc ~]$ cat my_blast.log
> Error memory mapping:/home/shams/ncbi-blast-2.9.0+/bin/nr.52.psq openedFilesCount=151 threadID=0
> Error: NCBI C++ Exception:
>     T0 "/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_560232_130.14.18.6_9008__PrepareRelease_Linux64-Centos_1552331742/c++/compilers/unix/../../src/corelib/ncbiobj.cpp", line 981: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to access NULL pointer.
>      Stack trace:
>       blastx ???:0 ncbi::CStackTraceImpl::CStackTraceImpl() offset=0x77 addr=0x1d95da7
>       blastx ???:0 ncbi::CStackTrace::CStackTrace(std::string const&) offset=0x25 addr=0x1d98465
>       blastx ???:0 ncbi::CException::x_GetStackTrace() offset=0xA0 addr=0x1ec7330
>       blastx ???:0 ncbi::CException::SetSeverity(ncbi::EDiagSev) offset=0x49 addr=0x1ec2169
>       blastx ???:0 ncbi::CObject::ThrowNullPointerException() offset=0x2D2 addr=0x1f42582
>       blastx ???:0 ncbi::blast::CBlastTracebackSearch::Run() offset=0x61C addr=0xf2929c
>       blastx ???:0 ncbi::blast::CLocalBlast::Run() offset=0x404 addr=0xed4684
>       blastx ???:0 CBlastxApp::Run() offset=0xC9C addr=0x9cbf7c
>       blastx ???:0 ncbi::CNcbiApplication::x_TryMain(ncbi::EAppDiagStream, char const*, int*, bool*) offset=0x8E3 addr=0x1da0e13
>       blastx ???:0 ncbi::CNcbiApplication::AppMain(int, char const* const*, char const* const*, ncbi::EAppDiagStream, char const*, std::string const&) offset=0x782 addr=0x1d9f6b2
>       blastx ???:0 main offset=0x5E5 addr=0x9caa05
>       /lib64/libc.so.6 ???:0 __libc_start_main offset=0xF5 addr=0x7f9a0fb3e505
>       blastx ???:0 blastx() [0x9ca345] offset=0x0 addr=0x9ca345
> 
> 
> 
> Any idea about that?
> 
> 
> Regards,
> Mahmood
> 
>