<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hmm.<br>
<br>
I've actually never used singleton and merely chained dependencies.
Maybe you can do that as a workaround:<br>
<br>
<i><b>waitjob=$(sbatch --hold fakejob.sh |sed 's/Submitted batch job
//')</b></i><i><b><br>
</b></i><i><b>for i in {1..2}; do</b></i><i><b><br>
</b></i><i><b> echo Submitting $VAULT to run after ${waitjob}</b></i><i><b><br>
</b></i><i><b> waitjob=$(sbatch -H --depend=afterany:${waitjob}
fakejob.sh|sed 's/Submitted batch job //')</b></i><i><b><br>
</b></i><i><b>done</b></i><br>
<p><br>
</p>
<p>Of course, if you are actually running the exact same script, I
would recommend using arrays as well.</p>
Brian Andrus<br>
<br>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 8/22/2019 6:23 AM, Jarno van der
Kolk wrote:<br>
</div>
<blockquote type="cite"
cite="mid:YTXPR0101MB16152DD0193504B48705A61DB0A50@YTXPR0101MB1615.CANPRD01.PROD.OUTLOOK.COM">
<pre class="moz-quote-pre" wrap="">Hi Brian,
Thanks for the suggestion. I tried it out, but I got the same result.
$ sbatch --hold --dependency=singleton ./fakejob.sh
Submitted batch job 26122715
$ sbatch --hold --dependency=singleton ./fakejob.sh
Submitted batch job 26122716
$ sbatch --hold --dependency=singleton ./fakejob.sh
Submitted batch job 26122720
$ scontrol update jobid=26122716 Dependency=singleton,after:26122720
$ scontrol update jobid=26122720 Dependency=singleton,after:26122715
$ scontrol release 26122715 26122716 26122720
... waiting for job 26122715 to complete ...
squeue -u jarno
JOBID USER ACCOUNT NAME ST TIME_LEFT NODES CPUS GRES MIN_MEM NODELIST (REASON)
26122716 jarno def-jarno_cp fakejob PD 2:00 1 1 (null) 250M (Dependency)
26122720 jarno def-jarno_cp fakejob PD 2:00 1 1 (null) 250M (Dependency)
Jarno van der Kolk, PhD Phys.
Analyste principal en informatique scientifique | Senior Scientific Computing Specialist
Solutions TI | IT Solutions
Université d’Ottawa | University of Ottawa
From: slurm-users <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users-bounces@lists.schedmd.com"><slurm-users-bounces@lists.schedmd.com></a> on behalf of Brian Andrus <a class="moz-txt-link-rfc2396E" href="mailto:toomuchit@gmail.com"><toomuchit@gmail.com></a>
Sent: August 21, 2019 5:26 PM
To: <a class="moz-txt-link-abbreviated" href="mailto:slurm-users@lists.schedmd.com">slurm-users@lists.schedmd.com</a> <a class="moz-txt-link-rfc2396E" href="mailto:slurm-users@lists.schedmd.com"><slurm-users@lists.schedmd.com></a>
Subject: Re: [slurm-users] Dependencies with singleton and after
Have you tried adding the dependency at submit time?
sbatch --dependency=singleton fakejob.sh
Brian Andrus
On 8/21/2019 1:51 PM, Jarno van der Kolk wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi,
I am helping a researcher who encountered an unexpected behaviour with dependencies. He uses both "singleton" and "after". he minimal working example is as follows:
$ sbatch --hold fakejob.sh
Submitted batch job 25909273
$ sbatch --hold fakejob.sh
Submitted batch job 25909274
$ sbatch --hold fakejob.sh
Submitted batch job 25909275
$ scontrol update jobid=25909273 Dependency=singleton
$ scontrol update jobid=25909274 Dependency=singleton,after:25909275
$ scontrol update jobid=25909275 Dependency=singleton,after:25909273
$ scontrol release 25909273 25909274 25909275
When releasing the jobs, the scheduler will start job 25909273 which is to be expected. The other jobs will be held due to the singleton and the jobs having the same job name, also expected.
However, when the job finishes, we would have expected job 25909275 to start since the singleton is now free and job 25909274 cannot start due to its dependency of "after:25909275". That is, the expected order would be 25909273 25909275 25909274 and one at a time.
Instead what happens is that job 25909273 starts and completes and then jobs 25909274 and 25909275 remain queued with unsatisfied dependencies.
It is entirely possible that I am thinking of this wrong of course, but I don't see it. Is this expected behaviour?
The content of fakejob.sh is simply this by the way, nothing special:
#!/bin/bash
#SBATCH --account=def-jarno
#SBATCH --time=0:1:30
#SBATCH --mem=250M
#SBATCH --ntasks=1
#SBATCH --job-name=fakejob
echo "Starting fake job"
sleep 60
echo "Finished fake job"
By the way, I realize this could be done with "afterany" instead of "singleton,after", but since this is a minimal working example it leaves out a lot of details of course.
Thanks,
Jarno
Jarno van der Kolk, PhD Phys.
Analyste principal en informatique scientifique | Senior Scientific Computing Specialist
Solutions TI | IT Solutions
Université d’Ottawa | University of Ottawa
</pre>
</blockquote>
</blockquote>
</body>
</html>