[slurm-users] Dependencies with singleton and after

[Jarno van der Kolk]

Hi,

I am helping a researcher who encountered an unexpected behaviour with dependencies. He uses both "singleton" and "after". The minimal working example is as follows:

$ sbatch --hold fakejob.sh
Submitted batch job 25909273
$ sbatch --hold fakejob.sh
Submitted batch job 25909274
$ sbatch --hold fakejob.sh
Submitted batch job 25909275
$ scontrol update jobid=25909273 Dependency=singleton
$ scontrol update jobid=25909274 Dependency=singleton,after:25909275
$ scontrol update jobid=25909275 Dependency=singleton,after:25909273
$ scontrol release 25909273 25909274 25909275

When releasing the jobs, the scheduler will start job 25909273 which is to be expected. The other jobs will be held due to the singleton and the jobs having the same job name, also expected.

However, when the job finishes, we would have expected job 25909275 to start since the singleton is now free and job 25909274 cannot start due to its dependency of "after:25909275". That is, the expected order would be 25909273 25909275 25909274 and one at a time.

Instead what happens is that job 25909273 starts and completes and then jobs 25909274 and 25909275 remain queued with unsatisfied dependencies.

It is entirely possible that I am thinking of this wrong of course, but I don't see it. Is this expected behaviour?

The content of fakejob.sh is simply this by the way, nothing special:
#!/bin/bash
#SBATCH --account=def-jarno
#SBATCH --time=0:1:30
#SBATCH --mem=250M
#SBATCH --ntasks=1
#SBATCH --job-name=fakejob

echo "Starting fake job"
sleep 60
echo "Finished fake job"


By the way, I realize this could be done with "afterany" instead of "singleton,after", but since this is a minimal working example it leaves out a lot of details of course.

Thanks,
Jarno

Jarno van der Kolk, PhD Phys.
Analyste principal en informatique scientifique | Senior Scientific Computing Specialist
Solutions TI | IT Solutions
Université d’Ottawa | University of Ottawa