<div dir="ltr"><div><br></div><div>Thank you all for such interesting replies.</div><div><br></div><div>The --dependency option is quite useful but in practice it has some inconvenients. Firstly, all 20 jobs are <u><b>instantly queued</b></u> which some users may be interpreting as an abusive use of common resources. Even worse, if a job fails, the rest one will stay queued forever (?) being the first tagged as "DependencyNeverSatisfied", and the rest just as "Dependency".<br></div><div><br></div><div>PS: Yarom, with queue time I meant the total run time allowed. I my case, after a job starts running it will be killed if it takes more than 10 hours of execution time. If the partition queue time limit were of 10 days for instance I guess I could use a single sbatch to launch an script containing the 20 executions as steps with srun<br></div><div><br></div><div>Regards,</div><div>Nigella<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El lun., 25 nov. 2019 a las 15:08, Yair Yarom (<<a href="mailto:irush@cs.huji.ac.il">irush@cs.huji.ac.il</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Hi,</div><div><br></div><div>I'm not sure what queue time limit of 10 hours is. If you can't have jobs waiting for more than 10 hours, than it seems to be very small for 8 hours jobs.<br></div><div></div><div>Generally, a few options:<br></div><div>a. The --dependency option (either afterok or singleton)</div><div>b. The --array option of sbatch with limit of 1 job at a time (instead of the for loop): sbatch --array=1-20%1 <br></div><div>c. At the end of the script of each job, call the sbatch line of the next job (this is probably the only option if indeed I understood the queue time limit correctly).<br></div></div><div><br></div><div>And indeed, srun should probably be reserved for strictly interactive jobs.<br></div><div><br></div><div>Regards,</div><div> Yair.</div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 25, 2019 at 11:21 AM Nigella Sanders <<a href="mailto:nigella.sanders@gmail.com" target="_blank">nigella.sanders@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>
<div>Hi all,<br></div><div><br></div><div>I guess this is a simple matter but I still find it confusing.</div><div><br></div><div>I have to run 20 jobs on our supercomputer. <br>Each job takes about 8 hours and every one need the previous one to be completed.<br></div><div>The queue time limit for jobs is 10 hours.</div><div><br></div><div>So my first approach is serially launching them in a loop using srun:</div><div><br></div><div><b><span style="font-family:monospace">#!/bin/bash<br></span></b></div><div><b><span style="font-family:monospace">for i in {1..20};do</span></b></div><div><b><span style="font-family:monospace"> srun --time 08:10:00 [options]<br></span></b></div><div><b><span style="font-family:monospace">done<br></span></b></div><div><b><span style="font-family:monospace"><br></span></b></div><div>However
SLURM literature keeps saying that 'srun' should be only used for short
command line tests. So that some sysadmins would consider this a bad
practice (<a href="https://stackoverflow.com/questions/43767866/slurm-srun-vs-sbatch-and-their-parameters" target="_blank">see this</a>).</div><div><br></div><div>My second approach switched to sbatch:</div><div><br></div><div>
<div><b><span style="font-family:monospace">
#!/bin/bash
</span></b></div><div><b><span style="font-family:monospace">for i in {1..20};do</span></b></div><div><b><span style="font-family:monospace"> sbatch
--time 08:10:00
[options]</span></b></div><div><b><span style="font-family:monospace"> [polling to queue to see if job is done]<br></span></b></div><div><b><span style="font-family:monospace">done</span></b><br></div>
</div><div><br></div><div>But since sbatch returns the prompt I had to
add code to check for job termination. Polling make use of sleep command
and it is prone to race conditions so it doesn't like to sysadmins
either.</div><div><br></div><div>I guess there must be a --wait option in some recent versions of SLURM (<a href="https://bugs.schedmd.com/show_bug.cgi?id=1685" target="_blank">see this</a>). Not yet available in our system though.<br></div><div><br></div><div>Is there any prefererable/canonical/friendly way to do this?<br></div>Any thoughts would be really appreciated,<br><div><br></div><div>Regards,<br></div><div>Nigella.<div><br></div></div></div><div><br>
</div></div>
</blockquote></div><br></div>
</blockquote></div>