<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">
<div class="gmail_default" style="font-family:tahoma,sans-serif">>If your SLURM version is at least 18.08 then you should be able to do it with an heterogeneous job. See <a href="https://slurm.schedmd.com/heterogeneous_jobs.html" target="_blank">https://slurm.schedmd.com/>heterogeneous_jobs.html</a> <br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">From the example in that page, I have written this<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">
<pre>#!/bin/bash<br>#SBATCH --job-name=myQE<br>#SBATCH --output=big-job<br>#SBATCH --mem-per-cpu=10g --ntasks=8<br>#SBATCH packjob<br>#SBATCH --mem-per-cpu=10g --ntasks=2<br>#SBATCH --partition=QUARTZ<br>#SBATCH --account=z5<br>mpirun pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
</pre>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">So, I expect that first node runs with 8 cores and 80GB of memory and the second node runs with 2 cores and 20GB or memory.</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">This is what I see in the output</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ squeue<br> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br> 723+0 CLUSTER myQE
mahmood
R 2:37 1 compute-0-1<br> 723+1 QUARTZ myQE
mahmood
R 2:37 1 compute-0-2<br>$ rocks run host compute-0-1 "ps aux | grep pw.x"<br>mahmood 25667 0.0 0.0 289544 13224 ? Sl 11:32 0:00 mpirun pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25672 99.4 2.0 3784972 2316076 ? Rl 11:32 2:49 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25673 99.4 2.0 3783544 2314008 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25674 99.4 2.0 3785532 2314196 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25675 99.2 2.0 3787648 2316048 ? Rl 11:32 2:49 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25676 99.4 2.0 3786600 2313916 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25677 99.4 2.0 3786344 2314056 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25678 99.4 2.0 3782632 2313892 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25679 99.4 2.0 3784112 2313856 ? Rl 11:32 2:50 pw.x -i <a href="http://mos2.rlx.in">mos2.rlx.in</a><br>
mahmood
25889 1.0 0.0 113132 1588 ? Ss 11:35 0:00 bash -c ps aux | grep pw.x<br>
mahmood
25925 0.0 0.0 112664 960 ? S 11:35 0:00 grep pw.x<br>$ rocks run host compute-0-2 "ps aux | grep pw.x"<br>
mahmood
28296 0.0 0.0 113132 1588 ? Ss 11:35 0:00 bash -c ps aux | grep pw.x<br>
mahmood
28325 0.0 0.0 112664 960 ? S 11:35 0:00 grep pw.x<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">So, compute-0-2 has no pw.x process.</div><div class="gmail_default" style="font-family:tahoma,sans-serif">Also, the partition names are weird. We have these entries:<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">$ cat /etc/slurm/parts<br>PartitionName=WHEEL RootOnly=yes Priority=1000 Nodes=ALL<br>PartitionName=RUBY AllowAccounts=y4,y8 Nodes=compute-0-[1-4]<br>PartitionName=EMERALD AllowAccounts=z2,z33,z7 Nodes=compute-0-[0-4],rocks7<br>PartitionName=QEMU AllowAccounts=q20_8 Nodes=compute-0-[1-4],rocks7<br>PartitionName=QUARTZ AllowAccounts=z5 Nodes=compute-0-[1-2],compute-0-4<br><br></div><div><br></div>
</div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div class="gmail_default" style="font-family:tahoma,sans-serif">Any thought?<br></div><div class="gmail_default" style="font-family:tahoma,sans-serif"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font face="tahoma,sans-serif">Regards,<br>Mahmood</font><br><br><br></div></div></div><br></div><br></div>