<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">>alternatively, you can ask slurm not to limit VSZ: in cgroup.conf, have<br>
>ConstrainSwapSpace=no<br>
>this does not actually permit arbitrary VSZ, since there are mechanisms<br>
>outside the cgroup limit that affect max VSZ (overcommit sysctls, swap space)</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Hi Mark,</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">ConstrainSwapSpace=no or ConstrainSwapSpace=yes has no effect...</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">
[root@hpc ~]$ cat /etc/slurm/cgroup.conf<br>CgroupAutomount=yes<br>CgroupReleaseAgentDir="/etc/slurm/cgroup"<br>ConstrainCores=no<br>ConstrainRAMSpace=no<br>ConstrainSwapSpace=no
</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[root@hpc ~]# systemctl restart slurmd<br>[root@hpc ~]# systemctl restart slurmdbd<br>[root@hpc ~]# systemctl restart slurmctld</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">
[root@hpc ~]# su - shams<br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[shams@hpc ~]$ cat slurm_blast.sh<br>#!/bin/bash<br>#SBATCH --job-name=blast1<br>#SBATCH --output=my_blast.log<br>#SBATCH --partition=SEA<br>#SBATCH --account=fish<br>#SBATCH --mem=38GB<br>#SBATCH --nodelist=hpc<br>#SBATCH --nodes=1<br>#SBATCH --ntasks-per-node=2<br>export PATH=~/ncbi-blast-2.9.0+/bin:$PATH<br>blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2<br>[shams@hpc ~]$ sbatch slurm_blast.sh<br>Submitted batch job 299<br>[shams@hpc ~]$ squeue<br> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br>[shams@hpc ~]$ cat my_blast.log<br>Error memory mapping:/home/shams/ncbi-blast-2.9.0+/bin/nr.52.psq openedFilesCount=151 threadID=0<br>Error: NCBI C++ Exception:<br> T0 "/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_560232_130.14.18.6_9008__PrepareRelease_Linux64-Centos_1552331742/c++/compilers/unix/../../src/corelib/ncbiobj.cpp", line 981: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to access NULL pointer.<br> Stack trace:<br> blastx ???:0 ncbi::CStackTraceImpl::CStackTraceImpl() offset=0x77 addr=0x1d95da7<br> blastx ???:0 ncbi::CStackTrace::CStackTrace(std::string const&) offset=0x25 addr=0x1d98465<br> blastx ???:0 ncbi::CException::x_GetStackTrace() offset=0xA0 addr=0x1ec7330<br> blastx ???:0 ncbi::CException::SetSeverity(ncbi::EDiagSev) offset=0x49 addr=0x1ec2169<br> blastx ???:0 ncbi::CObject::ThrowNullPointerException() offset=0x2D2 addr=0x1f42582<br> blastx ???:0 ncbi::blast::CBlastTracebackSearch::Run() offset=0x61C addr=0xf2929c<br> blastx ???:0 ncbi::blast::CLocalBlast::Run() offset=0x404 addr=0xed4684<br> blastx ???:0 CBlastxApp::Run() offset=0xC9C addr=0x9cbf7c<br> blastx ???:0 ncbi::CNcbiApplication::x_TryMain(ncbi::EAppDiagStream, char const*, int*, bool*) offset=0x8E3 addr=0x1da0e13<br> blastx ???:0 ncbi::CNcbiApplication::AppMain(int, char const* const*, char const* const*, ncbi::EAppDiagStream, char const*, std::string const&) offset=0x782 addr=0x1d9f6b2<br> blastx ???:0 main offset=0x5E5 addr=0x9caa05<br> /lib64/libc.so.6 ???:0 __libc_start_main offset=0xF5 addr=0x7f6119e0e505<br> blastx ???:0 blastx() [0x9ca345] offset=0x0 addr=0x9ca345<br>[shams@hpc ~]$ free -mh<br> total used free shared buff/cache available<br>Mem: 62G 9.2G 289M 277M 53G 52G<br>Swap: 9G 1.0M 9G<br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Please note that when I run the command</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[shams@hpc ~]$ blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2<br>
</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">The top command shows the following memory values</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>24449 shams 20 0 80.1g 1.3g 1.3g S 199.7 2.1 7:20.67 blastx</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">The Res value increases slightly over time but at the beginning of the run, the VIRT is 80GB.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Even, if my have specified small --mem (now 38GB), the error should be thrown in the middle of the run.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">However, when I use sbatch, the program quickly terminates.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span>
</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div><br></div>