<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Dear Colleagues,</p>
<p><br>
</p>
<p>already for quite some years now are we again and again facing issues on our clusters with so-called job-farming (or task-farming) concepts in Slurm jobs using srun. And it bothers me that we can hardly help users with requests in this regard.</p>
<p><br>
</p>
<p>From the documentation (<a href="https://slurm.schedmd.com/srun.html#SECTION_EXAMPLES" class="OWAAutoLink">https://slurm.schedmd.com/srun.html#SECTION_EXAMPLES</a>), it reads like this.</p>
<p><span>-------------------------------------------></span><br>
</p>
<p>...</p>
<p>#SBATCH --nodes=??</p>
<p>...</p>
<p>srun -N 1 -n 2 ... prog1 &> log.1 &<br>
</p>
<p><span>srun -N 1 -n 1 ... prog2 &> log.2 &</span><br>
</p>
<p>...</p>
<p>wait</p>
<p>-------------------------------------------></p>
<p>should do it, meaning that as many job steps are created and reasonably placed to the resources/slots available in the job allocation.</p>
<p><br>
</p>
<p>Well, this does not work so really on our clusters. (I'm afraid that I'm just too idiotic to use srun here ... )<br>
</p>
<p>As long as complete nodes are used, and regular task-per-node/cpus-per-task pattern, everything is still manageable. Task and thread placement using srun is sometimes still some burden.</p>
<p><br>
</p>
<p>But if I want rather free resource specifications, like in the example below (with half a node or so), I simply fail to get the desired result.<br>
</p>
<p><br>
</p>
<p>Ok. We've Haswell nodes with 2 sockets, and each socket has 2 NUMA domains with each 7 CPUs. 28 physical cores in total. 56 with Hyperthreading, such that the logical CPUs are as follows.</p>
<p>socket phys, CPU logic. CPU<br>
</p>
<p>0 0 0,28<br>
</p>
<p><span>0 1 1,29</span><br>
</p>
<p><span>0 2 2,30</span></p>
<p>...</p>
<p>1 0 14,42</p>
<p>...</p>
<p>1 13 27,55 <br>
</p>
<p>(slurm.conf is attached ... essential is "cm2_inter" partition of "inter" cluster)<br>
</p>
<p><br>
</p>
<p>So, for instance, for an OpenMP-only program, I'd like to place 14 OMP threads on the 1st socket, another step with 14 OMP threads on the 2nd socket (of first node), etc.<br>
</p>
<p><span>-------------------------------------------></span></p>
<p><span></p>
<div>#!/bin/bash<br>
#SBATCH -J jobfarm_test<br>
#SBATCH -o log.%x.%j.%N.out<br>
#SBATCH -D ./<br>
#SBATCH --mail-type=NONE<br>
#SBATCH --time=00:10:00<br>
#SBATCH --export=NONE<br>
#SBATCH --get-user-env<br>
#SBATCH --clusters=cm2<br>
#SBATCH --partition=cm2_std<br>
#SBATCH --nodes=2<br>
module load slurm_setup # specific to LRZ cluster<br>
<br>
export OMP_NUM_THREADS=14<br>
placement=/lrz/sys/tools/placement_test_2021/bin/placement-test.omp_only<br>
srun_opts="-N 1 -n 1 -c $((2*$OMP_NUM_THREADS)) --mpi=none --exclusive --export=ALL --cpu_bind=verbose,cores"<br>
<br>
for i in $(seq 1 4); do<br>
srun $srun_opts $placement -d 10 &> log.$i &<br>
done <br>
wait<span></span><br>
<span></span></div>
</span>
<p></p>
<p><span>-------------------------------------------></span><br>
</p>
<p>placemen-test.omp_only is just an OMP executable, where each thread reads the /proc/... info about ttid and cpu it is running on an prints it screen (and the sleeps in order to persist longer on the cpu in running state).</p>
<p><br>
</p>
<p>With the script above, I assumed that this would let run all 4 srun-steps at the same time - on each socket one. But it doesn't.</p>
<p><br>
</p>
<p>First of all, due to Hyptherthreading, I must specify here "-c 56". If I would use "-c 28" (which would be more intuitive to me), the CPUs 0-6,28-32 are used (the first NUMA domain). And also then if I use -c 28 or even -c 14, the steps don't run at the
same time on a node. Only a single step per node at a time.</p>
<p><br>
</p>
<p>Removing "--exclusive" doesn't change anything. --cpu_bind to socket doesn't have an effect either (here I already shoot into the blue).</p>
<p><br>
</p>
<p>I want to avoid more stringent requirements (something like memory), as to admit sharing the complete available memory on a node. But even if I reduced memory requirement per CPU (ridiculous 1K) does not change anything.</p>
<p><br>
</p>
<p><br>
</p>
<p>So something I definitely do wrong. But I can't even guess, what? I tried really many options. Also -m and -B options. Without success. Complexity is killing me here.<br>
</p>
<p>From SchedMD documentation, I assume it shouldn't be so complicated as to use the low level placement options</p>
<div> - as they described here <a href="https://slurm.schedmd.com/mc_support.html" class="OWAAutoLink">
https://slurm.schedmd.com/mc_support.html</a> </div>
<p></p>
<p><br>
</p>
<p><br>
</p>
<p>Has someone any some clue on how to use srun for these purposes?</p>
<p>But if not, I would also be glad to learn about alternatives ... if they are as convenient as SchedMD promised with
<a href="https://slurm.schedmd.com/srun.html#SECTION_EXAMPLES" class="OWAAutoLink">
https://slurm.schedmd.com/srun.html#SECTION_EXAMPLES</a></p>
<p>Then I can get rid of srun ... maybe. (In my desperation, I even tried GNU parallel with SSH process spawning ... :( Fazit: It is not really convenient for that purpose.)<br>
</p>
<p><br>
</p>
<p>Thank you very much in advance!</p>
<p>Kind regards,</p>
<p>Martin<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</div>
</body>
</html>