<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">Hi,</div><div class="gmail_default" style="font-family:tahoma,sans-serif">I used to run a hello mpi for testing purposes. Now, I see that it doesn't work. While the log file shows memory allocation problem, squeue shows that job is in R state endlessly.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">[mahmood@hpc ~]$ cat slurm_script1.sh<br>#!/bin/bash<br>#SBATCH --job-name=hello_mpi<br>#SBATCH --output=hellompi.log<br>#SBATCH --ntasks=4<br>#SBATCH --time=10:00<br>#SBATCH --partition=SEA<br>#SBATCH --account=fish<br>#SBATCH --mem=100M<br>mpirun ./mpihello<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">[mahmood@hpc ~]$ sbatch slurm_script1.sh<br>Submitted batch job 10<br>[mahmood@hpc ~]$ cat hellompi.log<br>[<a href="http://hpc.safaar.com:18059">hpc.safaar.com:18059</a>] create_and_attach: unable to create shared memory BTL coordinating structure :: size 134217728<br>--------------------------------------------------------------------------<br>A system call failed during shared memory initialization that should<br>not have.  It is likely that your MPI job will now either abort or<br>experience performance degradation.<br><br>  Local host:  <a href="http://hpc.safaar.com">hpc.safaar.com</a><br>  System call: mmap(2)<br>  Error:       Cannot allocate memory (errno 12)<br>--------------------------------------------------------------------------<br>[mahmood@hpc ~]$ squeue<br>             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)<br>                10       SEA hello_mp  mahmood  R       0:36      1 hpc<br>[mahmood@hpc ~]$ squeue<br>             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)<br>                10       SEA hello_mp  mahmood  R       0:43      1 hpc</div><div class="gmail_default" style="font-family:tahoma,sans-serif">[mahmood@hpc ~]$ sacctmgr list association format=partition,account,user,grptres<br> Partition    Account       User       GrpTRES<br>---------- ---------- ---------- -------------<br>                 root<br>                 root       root<br>                 fish<br>       sea       fish    mahmood cpu=10,mem=8G<br>                local    mahmood</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">However, the binary file works fine outside of slurm.</div><br><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div></div>