<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:14pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi <span style="color: rgb(33, 33, 33); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 18.6667px; background-color: rgb(255, 255, 255);">Alfredo</span>,<br>
</p>
<p><br>
</p>
<p>Beyond what is already suggested, I have used the following script to run a set number of jobs simultaneously within a batch script.<br>
</p>
<p><br>
</p>
<p></p>
<div><span style="font-family: Consolas, monospace; font-size: 10pt;">for thing in ${arrayOfThings[@]}; do echo $<span style="font-family: Consolas, monospace; font-size: 13.3333px; background-color: rgb(255, 255, 255);"><span style="font-family: Consolas, monospace; font-size: 13.3333px; background-color: rgb(255, 255, 255);">thing</span></span>;
done | (</span></div>
<div><span style="font-size: 10pt;"><span style="white-space: pre; font-family: Consolas, monospace;"></span><span style="font-family: Consolas, monospace;">srun -J JobName xargs -I{} --max-procs ${SLURM_JOB_CPUS_PER_NODE} bash -c '{</span></span></div>
<div><span style="font-size: 10pt;"><span style="white-space: pre; font-family: Consolas, monospace;"></span><span style="font-family: Consolas, monospace;">someCommand -args -and -options {}</span></span></div>
<div><span style="font-size: 10pt;"><span style="white-space: pre; font-family: Consolas, monospace;"></span><span style="font-family: Consolas, monospace;">}'</span></span></div>
<div><span style="font-family: Consolas, monospace; font-size: 10pt;">)</span></div>
<div><span style="font-family: Consolas, monospace; font-size: 10pt;"><br>
</span></div>
<p>Where {} is $thing<br>
</p>
<p><br>
</p>
<p>In operation <a href="https://github.com/shawarden/Fastq2VCF/blob/master/slurm-scripts/blockalign.sl#L215">
here</a><br>
</p>
<p><br>
</p>
<p>Cheers,<br>
</p>
<p> Sam<br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div class="BodyFragment">
<hr style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:center">
<div class="PlainText">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right"><span style="font-size:large">Sam Hawarden</span></div>
<font size="4">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Assistant Research Fellow</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Pathology Department</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">Dunedin School of Medicine</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; text-align:right">sam.hawarden(at)otago.ac.nz<br>
</div>
<div style="text-align:right"></div>
</font></div>
</div>
</div>
</div>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> slurm-users <slurm-users-bounces@lists.schedmd.com> on behalf of Carlos Fenoy <minibit@gmail.com><br>
<b>Sent:</b> Thursday, 20 December 2018 04:59<br>
<b>To:</b> Slurm User Community List<br>
<b>Subject:</b> Re: [slurm-users] requesting resources and afterwards launch an array of calculations</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">Hi Alfredo,
<div><br>
</div>
<div>You can have a look at using <a href="https://github.com/eth-cscs/GREASY">https://github.com/eth-cscs/GREASY</a> . It was developed before array-jobs were supported in slurm and it will do exactly what you want.</div>
<div><br>
</div>
<div>Regards,<br>
Carlos</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Dec 19, 2018 at 3:33 PM Alfredo Quevedo <<a href="mailto:maquevedo.unc@gmail.com">maquevedo.unc@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
thank you Michael for the feedback, my scenario is the following: I want <br>
to run a job array of (lets say) 30 jobs. So I setted the slurm input as <br>
follows:<br>
<br>
#SBATCH --array=1-104%30<br>
#SBATCH --ntasks=1<br>
<br>
however only 4 jobs within the array are launched at a time due to the <br>
allowed max number of jobs as setted in the slurm configuration (4). As <br>
a workaround to the issued, the sysadmin suggested me to request the <br>
resources, and afterwards distribute the resources asigned into a <br>
multiple set of single CPU task. I believe that with the solution you <br>
mentioned only 30 (out of the 104) jobs will be finished?<br>
<br>
thanks<br>
<br>
Alfredo<br>
<br>
<br>
El 19/12/2018 a las 11:15, Renfro, Michael escribió:<br>
> Literal job arrays are built into Slurm: <a href="https://slurm.schedmd.com/job_array.html" rel="noreferrer" target="_blank">
https://slurm.schedmd.com/job_array.html</a><br>
><br>
> Alternatively, if you wanted to allocate a set of CPUs for a parallel task, and then run a set of single-CPU tasks in the same job, something like:<br>
><br>
> #!/bin/bash<br>
> #SBATCH --ntasks=30<br>
> srun --ntasks=${SLURM_NTASKS} hostname<br>
><br>
> is one way of doing it. If that’s not what you’re looking for, some other details would be needed.<br>
><br>
<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">--<br>
Carles Fenoy<br>
</div>
</div>
</div>
</body>
</html>