[slurm-users] Managing Multiple Dependencies

[Jan van der Laan]

Snakemake is a tool to manage job dependencies. I don't have experience 
using this in a slurm context, but I have used Snakemake to manage 
complex depencies in workflows in other contexts. Snakemake should 
support slurm.

HTH,
Jan


On 02-03-2021 20:16, Jason Simms wrote:
> Hello all,
> 
> I am relatively new to the nuances of handling complex dependencies in 
> Slurm, so I'm hoping the hive mind can help. I have a user wanting to 
> accomplish the following:
> 
>   * submit one job
>   * submit multiple jobs that are dependent on the output from the first
>     job (so they just need to launch once the first job has completed)
>   * submit one final job dependent on all the previous jobs completing
> 
> Is there a way to do this cleanly? So it's a three stage process. I have 
> ideas in my head of writing Slurm JobIDs to a file, reading them out, 
> and managing dependencies that way, but perhaps there is a more 
> efficient way (or perhaps not!).
> 
> Warmest regards,
> Jason
> 
> -- 
> *Jason L. Simms, Ph.D., M.P.H.*
> Manager of Research and High-Performance Computing
> XSEDE Campus Champion
> Lafayette College
> Information Technology Services
> 710 Sullivan Rd | Easton, PA 18042
> Office: 112 Skillman Library
> p: (610) 330-5632