<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:courier new,monospace">If you want to have the same number of processes per node, like:</div><div class="gmail_default" style="font-family:courier new,monospace"><br></div><div class="gmail_default" style=""><blockquote style="font-family:"courier new",monospace;margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_default" style="font-family:courier new,monospace">#PBS -l nodes=4:ppn=8<br></div><br></blockquote><span style="font-family:"courier new",monospace">then what I am doing (maybe there is another way?) is:</span></div><div class="gmail_default" style=""><font face="courier new, monospace"><br></font><blockquote style="font-family:"courier new",monospace;margin:0px 0px 0px 40px;border:none;padding:0px">#SBATCH --ntasks-per-node=8 <br>#SBATCH --nodes=4 <br>#SBATCH --mincpus=8 <br><br></blockquote><font face="courier new, monospace">This is because "--ntasks-per-node" is actually "maximum number of tasks per node" and "--nodes</font><span style="font-family:"courier new",monospace">=4" means "minimum number of nodes". I'm sure other variations (specifying --ntasks=32, --mincpus=8 and --nodes=4-4 might do it too) but this one is what I've been using. I remember being surprised when coming over from Torque to find that "--ntasks-per-node" and --nodes did not mean what they so obviously seemed to mean.</span></div><div class="gmail_default" style=""><blockquote style="font-family:"courier new",monospace;margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_default" style="font-family:courier new,monospace"><br></div></blockquote><font face="courier new, monospace">Steve</font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 24, 2022 at 7:56 PM David Henkemeyer <<a href="mailto:david.henkemeyer@gmail.com">david.henkemeyer@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Thank you! We recently converted from pbs, and I was converting “ppn=X” to “-n X”. Does it make more sense to convert “ppn=X” to --“cpus-per-task=X”?</div><div dir="auto"><br></div><div dir="auto">Thanks again</div><div dir="auto">David</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 24, 2022 at 3:54 PM Thomas M. Payerle <<a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Although all three cases ( "-N 1 --cpus-per-task 64 -n 1", "-N 1 --cpus-per-task 1 -n 64", and "-N 1 --cpus-per-task 32 -n 2") will cause Slurm to allocate 64 cores to the job, there can (and will) be differences in the other respects.</div><div><br></div><div>The variable SLURM_NTASKS will be set to the argument of the -n (aka --ntasks) argument, and other Slurm variables will differ as well.</div><div><br></div><div>More importantly, as others noted, srun will launch $SLURM_NTASKS processes. The mpirun/mpiexec/etc binaries of most MPI libraries will (if compiled with support for Slurm) act similarly (and indeed, I believe most use srun under the hood).</div><div><br></div><div>If you are just using sbatch and launching a single process using 64 threads, then the different options are probably equivalent for most intent and purposes. Similar if you are doing a loop to start 64 single threaded processes. But those are simplistic cases, and just happen to "work" even though you are "abusing" the scheduler options. And even the cases wherein it "works" is subject to unexpected failures (e.g. if one substitutes srun for sbatch). <br></div><div><br></div><div>The differences are most clear when the -N 1 flag is not given. Generally, SLURM_NTASKS should be the number of MPI or similar tasks you intend to start. By default, it is assumed the tasks can support distributed memory parallelism, so the scheduler by default assumes that it can launch tasks on different nodes (the -N 1 flag you mentioned would override that). Each such task is assumed to need --cpus-per-task cores which the scheduler assumes needs shared memory parallelism (i.e. must be on the same node). <br></div><div>So without the -N 1, "--cpus-per-task 64 -n 1" will require 64 cores on a single node, whereas "-n 64 --cpus-per-task 1" can result in the job being assigned 64 cores on a single node to a single core on 64 nodes or any combination in between with 64 cores. The "--cpus-per-task 32 -n 2" will either assign one node with 64 cores or 2 nodes with 32 cores each.</div><div><br></div><div>As I said, although there are some simple cases where the different cases are mostly functionally equivalent, I would recommend trying to use the proper arguments --- "abusing" the arguments might work for a while but will likely bite you in the end. E.g., the 64 thread case should do "--cpus-per-task 64", and the launching processes in the loop should _probably_ do "-n 64" (assuming it can handle the tasks being assigned to different nodes).<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 24, 2022 at 3:35 PM David Henkemeyer <<a href="mailto:david.henkemeyer@gmail.com" target="_blank">david.henkemeyer@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Assuming -N is 1 (meaning, this job needs only one node), then is there a difference between any of these 3 flag combinations:<div dir="auto"><br></div><div dir="auto">-n 64 (leaving cpus-per-task to be the default of 1)</div><div dir="auto"><span style="color:rgb(0,0,0)">--cpus-per-task</span><span style="color:rgb(0,0,0)"> </span>64 (leaving -n to be the default of 1)</div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)">--cpus-per-task 32 -n 2</span><br></div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)"><br></span></div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)">As far as I can tell, there is no functional difference. But if there is even a subtle difference, I would love to know what it is!</span></div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)"><br></span></div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)">Thanks</span></div><div dir="auto"><span style="border-color:rgb(0,0,0);color:rgb(0,0,0)">David </span></div>-- <br><div dir="ltr">Sent from Gmail Mobile</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Tom Payerle <br>DIT-ACIGS/Mid-Atlantic Crossroads <a href="mailto:payerle@umd.edu" target="_blank">payerle@umd.edu</a><br></div><div>5825 University Research Park (301) 405-6135<br></div><div dir="ltr">University of Maryland<br>College Park, MD 20740-3831<br></div></div></div></div></div></div>
</blockquote></div></div>-- <br><div dir="ltr">Sent from Gmail Mobile</div>
</blockquote></div><div><br></div></div>