<div dir="ltr"><div class="gmail_default" style="font-size:small">I would suggest using Gnu Parallel (<a href="https://www.gnu.org/software/parallel/">https://www.gnu.org/software/parallel/</a>). Also, if you run that many "srun" in a row, on a very large cluster where the slurmctl is very solicited some of the srun might time out and not run.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Richard </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 5 nov. 2021 à 05:45, Marcus Pedersén <<a href="mailto:marcus.pedersen@slu.se">marcus.pedersen@slu.se</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi all,<br>
I have setup a basic slurm system and been testing out<br>
a nuber of things.<br>
The latest thing I started to test is the parallel parts.<br>
What I have is about 70 independent scripts that would be<br>
ideal to run in parallel.<br>
For testing purposes I have created 20 dummy scripts<br>
that print script name, hostname sleeps for one minute<br>
and prints no of minutes.<br>
<br>
The way I want to run this is to allocate 2 nodes<br>
and run all of the 20 scripts in parallel, each one of them<br>
in one process.<br>
My idea is that the first node will be filled up with 12 processes,<br>
each process running one script and the second node will run<br>
the rest of the processes/scripts (8 scripts on 8 processes).<br>
I have read up on a couple of tutorials and looked at the documentation<br>
for different parts of slurm.<br>
But what ever flags I use for both sbatch and srun I do not seem to<br>
be able to accomplish what I want.<br>
All nodes have 6 cores with 2 threads.<br>
<br>
The closest I have come is with this small sbatch:<br>
<br>
#! /bin/bash<br>
#SBATCH --job-name=TestParallel<br>
#SBATCH --nodes=2<br>
#SBATCH --ntasks-per-node=1<br>
#SBATCH --ntasks=2<br>
#SBATCH --cpus-per-task=12<br>
#SBATCH --nodelist=node1,node2<br>
#SBATCH --output="%x-%4j-%N.out"<br>
#SBATCH --mail-user=my@mail<br>
#SBATCH --mail-type=ALL<br>
<br>
echo<br>
date +%Y-%m-%d" "%H-%M-%S<br>
<br>
for i in {1..20}<br>
do<br>
srun --nodes=1 --ntasks=1 --ntasks-per-node=1 --cpus-per-task=1 --exclusive --job-name=Testp-$i --output=/path/to/test_prog$i.log /path/to/test_prog$i.sh &<br>
done<br>
<br>
date +%Y-%m-%d" "%H-%M-%S<br>
<br>
wait<br>
<br>
<br>
sacct gives the following output:<br>
505 TestParal+ all marcus 24 RUNNING node[1-2] 0:0<br>
505.batch batch 12 RUNNING node1 0:0<br>
505.0 Testp-3 1 RUNNING node1 0:0<br>
505.1 Testp-6 1 RUNNING node2 0:0<br>
505.2 Testp-2 1 RUNNING node1 0:0<br>
505.3 Testp-13 1 RUNNING node1 0:0<br>
505.4 Testp-9 1 RUNNING node1 0:0<br>
505.5 Testp-11 1 RUNNING node1 0:0<br>
505.6 Testp-16 1 RUNNING node1 0:0<br>
505.7 Testp-12 1 RUNNING node1 0:0<br>
505.8 Testp-20 1 RUNNING node1 0:0<br>
505.9 Testp-4 1 RUNNING node1 0:0<br>
505.10 Testp-19 1 RUNNING node1 0:0<br>
505.11 Testp-10 1 RUNNING node1 0:0<br>
505.12 Testp-5 1 RUNNING node1 0:0<br>
<br>
<br>
Slurm only use one process on node2 and of cause I want all the last 8 processes to run on node2.<br>
<br>
I have tried a number of other options usualy ending in running the same script multiple times<br>
and that is not what I want.<br>
<br>
I feel a bit stuck and can not get my head around this.<br>
<br>
I would really appreciate some help!!<br>
<br>
Many thanks in advance!!<br>
<br>
Best Regards<br>
Marcus<br>
<br>
---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" rel="noreferrer" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" rel="noreferrer" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
<br>
</blockquote></div>