Hello
I have the following scenario: I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed).
If I just submit all those jobs via a loop using dependencies, then I end up with a lot of pending jobs who might later not even run because no output file has been produced by the preceding jobs. Is there a way to pause the submission loop until the required file has been generated so that at most two jobs are submitted at the same time?
Here is a sample submission script showing what I want to achieve.
for i in {1..200}; do FILE=GHM_paramset_${i}.dat # How can I pause the submission loop until the FILE has been created???? #if test -f "$FILE"; then jobid4=$(sbatch --parsable --dependency=afterok:$jobid3 job4_sub $i) jobid3=$(sbatch --parsable --dependency=afterok:$jobid4 job3_sub $i) #fi done
Any help will be appreciated
Amjad
Amjad Syed via slurm-users slurm-users@lists.schedmd.com writes:
I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed).
How about letting each even job submit the next odd job after it has created the file, and also the following even job, with a dependency on the odd job? You would obviuosly have to keep track of how many jobs you've submitted so you can stop after 400 jobs. :)
Hi Ajad,
Amjad Syed via slurm-users slurm-users@lists.schedmd.com writes:
Hello
I have the following scenario: I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed).
If I just submit all those jobs via a loop using dependencies, then I end up with a lot of pending jobs who might later not even run because no output file has been produced by the preceding jobs. Is there a way to pause the submission loop until the required file has been generated so that at most two jobs are submitted at the same time?
Here is a sample submission script showing what I want to achieve.
for i in {1..200}; do FILE=GHM_paramset_${i}.dat # How can I pause the submission loop until the FILE has been created???? #if test -f "$FILE"; then jobid4=$(sbatch --parsable --dependency=afterok:$jobid3 job4_sub $i) jobid3=$(sbatch --parsable --dependency=afterok:$jobid4 job3_sub $i) #fi done
Any help will be appreciated
Amjad
You might find a job array useful for this (for any large number of jobs with identical resources, using a job array also helps backfilling to work efficiently, if you are using it).
With a job array you can specify how many jobs should run simultaneously with the '%' notation:
--array=1-200%2
Cheers,
Loris
-
Amjad Syed -
Bjørn-Helge Mevik -
Loris Bennett