<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">OK. The two different partitions I saw was due to not specifying partition name for the first set (before packjob). Here is a better script</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">#!/bin/bash<br>#SBATCH --job-name=myQE<br>#SBATCH --output=big-mem<br>#SBATCH --mem-per-cpu=16g --ntasks=2<br>#SBATCH -N 1<br>#SBATCH --partition=QUARTZ<br>#SBATCH --account=z5<br>#SBATCH packjob<br>#SBATCH --mem-per-cpu=10g --ntasks=4<br>#SBATCH -N 1<br>#SBATCH --partition=QUARTZ<br>#SBATCH --account=z5<br>srun pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">One node should run 2 processes (32GB total) and one other node should run 4 process (40GB total).</div><div class="gmail_default" style="font-family:tahoma,sans-serif">The queue looks like</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ squeue<br> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br> 747+1 QUARTZ myQE ghatee R 0:02 1 rocks7<br> 747+0 QUARTZ myQE ghatee R 0:02 1 compute-0-2<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">As I checked the node, one 2 processes are running on compute-0-2 (first set before packjob). But there is no processes on rocks7.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ rocks run host compute-0-2 "ps aux | grep pw.x"<br>ghatee 30234 0.0 0.0 251208 4996 ? Sl 15:04 0:00 srun pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>ghatee 30235 0.0 0.0 46452 748 ? S 15:04 0:00 srun pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>ghatee 30247 99.8 0.1 1930484 129696 ? Rl 15:04 4:31 /home/ghatee/QuantumEspresso621/bin/pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>ghatee 30248 99.8 0.1 1930488 129704 ? Rl 15:04 4:31 /home/ghatee/QuantumEspresso621/bin/pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>ghatee 30352 0.0 0.0 113132 1592 ? Ss 15:09 0:00 bash -c ps aux | grep pw.x<br>ghatee 30381 0.0 0.0 112664 960 ? S 15:09 0:00 grep pw.x<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ rocks run host rocks7 "ps aux | grep pw.x"<br>ghatee 17141 0.0 0.0 316476 26632 pts/21 Sl+ 23:39 0:00 /opt/rocks/bin/python /opt/rocks/bin/rocks run host rocks7 ps aux | grep pw.x<br>ghatee 17143 0.0 0.0 113132 1364 pts/21 S+ 23:39 0:00 bash -c ps aux | grep pw.x<br>ghatee 17145 0.0 0.0 112664 960 pts/21 R+ 23:39 0:00 grep pw.x<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Any idea?</div><div class="gmail_default" style="font-family:tahoma,sans-serif">It seems that the mpirun I have is not compatible with the hetro configuration because the SBATCH parameters are straight forward.<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br clear="all"></div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 27, 2019 at 11:03 PM Christopher Samuel <<a href="mailto:chris@csamuel.org">chris@csamuel.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 3/27/19 11:29 AM, Mahmood Naderan wrote:<br>
<br>
> Thank you very much. you are right. I got it.<br>
<br>
Cool, good to hear.<br>
<br>
I'd love to hear whether you get heterogenous MPI jobs working too!<br>
<br>
All the best,<br>
Chris<br>
-- <br>
Chris Samuel : <a href="http://www.csamuel.org/" rel="noreferrer" target="_blank">http://www.csamuel.org/</a> : Berkeley, CA, USA<br>
<br>
</blockquote></div></div>