<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">>alternatively, you can ask slurm not to limit VSZ: in cgroup.conf, have<br>
>ConstrainSwapSpace=no<br>
>this does not actually permit arbitrary VSZ, since there are mechanisms<br>
>outside the cgroup limit that affect max VSZ (overcommit sysctls, swap space)</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Hi Mark,</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">ConstrainSwapSpace=no or ConstrainSwapSpace=yes has no effect...</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">
[root@hpc ~]$ cat /etc/slurm/cgroup.conf<br>CgroupAutomount=yes<br>CgroupReleaseAgentDir="/etc/slurm/cgroup"<br>ConstrainCores=no<br>ConstrainRAMSpace=no<br>ConstrainSwapSpace=no
</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[root@hpc ~]# systemctl restart slurmd<br>[root@hpc ~]# systemctl restart slurmdbd<br>[root@hpc ~]# systemctl restart slurmctld</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">
[root@hpc ~]# su - shams<br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[shams@hpc ~]$ cat slurm_blast.sh<br>#!/bin/bash<br>#SBATCH --job-name=blast1<br>#SBATCH --output=my_blast.log<br>#SBATCH --partition=SEA<br>#SBATCH --account=fish<br>#SBATCH --mem=38GB<br>#SBATCH --nodelist=hpc<br>#SBATCH --nodes=1<br>#SBATCH --ntasks-per-node=2<br>export PATH=~/ncbi-blast-2.9.0+/bin:$PATH<br>blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2<br>[shams@hpc ~]$ sbatch slurm_blast.sh<br>Submitted batch job 299<br>[shams@hpc ~]$ squeue<br>       JOBID PARTITION   NAME   USER ST    TIME  NODES NODELIST(REASON)<br>[shams@hpc ~]$ cat my_blast.log<br>Error memory mapping:/home/shams/ncbi-blast-2.9.0+/bin/nr.52.psq openedFilesCount=151 threadID=0<br>Error: NCBI C++ Exception:<br>  T0 "/home/coremake/release_build/build/PrepareRelease_Linux64-Centos_JSID_01_560232_130.14.18.6_9008__PrepareRelease_Linux64-Centos_1552331742/c++/compilers/unix/../../src/corelib/ncbiobj.cpp", line 981: Critical: ncbi::CObject::ThrowNullPointerException() - Attempt to access NULL pointer.<br>   Stack trace:<br>   blastx ???:0 ncbi::CStackTraceImpl::CStackTraceImpl() offset=0x77 addr=0x1d95da7<br>   blastx ???:0 ncbi::CStackTrace::CStackTrace(std::string const&) offset=0x25 addr=0x1d98465<br>   blastx ???:0 ncbi::CException::x_GetStackTrace() offset=0xA0 addr=0x1ec7330<br>   blastx ???:0 ncbi::CException::SetSeverity(ncbi::EDiagSev) offset=0x49 addr=0x1ec2169<br>   blastx ???:0 ncbi::CObject::ThrowNullPointerException() offset=0x2D2 addr=0x1f42582<br>   blastx ???:0 ncbi::blast::CBlastTracebackSearch::Run() offset=0x61C addr=0xf2929c<br>   blastx ???:0 ncbi::blast::CLocalBlast::Run() offset=0x404 addr=0xed4684<br>   blastx ???:0 CBlastxApp::Run() offset=0xC9C addr=0x9cbf7c<br>   blastx ???:0 ncbi::CNcbiApplication::x_TryMain(ncbi::EAppDiagStream, char const*, int*, bool*) offset=0x8E3 addr=0x1da0e13<br>   blastx ???:0 ncbi::CNcbiApplication::AppMain(int, char const* const*, char const* const*, ncbi::EAppDiagStream, char const*, std::string const&) offset=0x782 addr=0x1d9f6b2<br>   blastx ???:0 main offset=0x5E5 addr=0x9caa05<br>   /lib64/libc.so.6 ???:0 __libc_start_main offset=0xF5 addr=0x7f6119e0e505<br>   blastx ???:0 blastx() [0x9ca345] offset=0x0 addr=0x9ca345<br>[shams@hpc ~]$ free -mh<br>       total     used     free    shared  buff/cache  available<br>Mem:       62G     9.2G     289M     277M     53G     52G<br>Swap:       9G     1.0M      9G<br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Please note that when I run the command</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">[shams@hpc ~]$ blastx -db ~/ncbi-blast-2.9.0+/bin/nr -query ~/khTrinityfilterless1.fasta -max_target_seqs 5 -outfmt 6 -evalue 1e-5 -num_threads 2<br>
</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">The top command shows the following memory values</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"> PID USER    PR  NI   VIRT   RES   SHR S  %CPU %MEM   TIME+ COMMAND<br>24449 shams   20  0  80.1g  1.3g  1.3g S 199.7  2.1  7:20.67 blastx</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">The Res value increases slightly over time but at the beginning of the run, the VIRT is 80GB.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">Even, if my have specified small --mem (now 38GB), the error should be thrown in the middle of the run.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im">However, when I use sbatch, the program quickly terminates.</span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><span class="gmail-im"><br></span>
</div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div><br></div>