I have the following scenario:
I need to submit a sequence of up to 400 jobs where the even jobs depend on the preceeding odd job to finish and every odd job depends on the presence of a file generated by the preceding even job (availability of the file for the first of those 400 jobs is guaranteed).
If I just submit all those jobs via a loop using dependencies, then I end up with a lot of pending jobs who might later not even run because no output file has been produced by the preceding jobs. Is there a way to pause the submission loop until the required file has been generated so that at most two jobs are submitted at the same time?
Here is a sample submission script showing what I want to achieve.
for i in {1..200}; do
FILE=GHM_paramset_${i}.dat
# How can I pause the submission loop until the FILE has been created????
#if test -f "$FILE"; then
jobid4=$(sbatch --parsable --dependency=afterok:$jobid3 job4_sub $i)
jobid3=$(sbatch --parsable --dependency=afterok:$jobid4 job3_sub $i)
#fi
done