[slurm-users] Managing Multiple Dependencies

[Jason Simms]

Dear Mike and Jan,

Thank you for your suggestions. Mike, your script was close to what I had
in mind, though way more elegant. So I appreciate it!

Also Jan, thank you for pointing out snakemake. Apparently people do use it
with Slurm, so I'm looking into it. Probably overkill at this point for
most of our users, but it's nice to know that it's an option.

Warmest regards,
Jason

On Tue, Mar 2, 2021 at 4:36 PM Jan van der Laan <slurm at eoos.dds.nl> wrote:

>
> Snakemake is a tool to manage job dependencies. I don't have experience
> using this in a slurm context, but I have used Snakemake to manage
> complex depencies in workflows in other contexts. Snakemake should
> support slurm.
>
> HTH,
> Jan
>
>
> On 02-03-2021 20:16, Jason Simms wrote:
> > Hello all,
> >
> > I am relatively new to the nuances of handling complex dependencies in
> > Slurm, so I'm hoping the hive mind can help. I have a user wanting to
> > accomplish the following:
> >
> >   * submit one job
> >   * submit multiple jobs that are dependent on the output from the first
> >     job (so they just need to launch once the first job has completed)
> >   * submit one final job dependent on all the previous jobs completing
> >
> > Is there a way to do this cleanly? So it's a three stage process. I have
> > ideas in my head of writing Slurm JobIDs to a file, reading them out,
> > and managing dependencies that way, but perhaps there is a more
> > efficient way (or perhaps not!).
> >
> > Warmest regards,
> > Jason
> >
> > --
> > *Jason L. Simms, Ph.D., M.P.H.*
> > Manager of Research and High-Performance Computing
> > XSEDE Campus Champion
> > Lafayette College
> > Information Technology Services
> > 710 Sullivan Rd | Easton, PA 18042
> > Office: 112 Skillman Library
> > p: (610) 330-5632
>
>

-- 
*Jason L. Simms, Ph.D., M.P.H.*
Manager of Research and High-Performance Computing
XSEDE Campus Champion
Lafayette College
Information Technology Services
710 Sullivan Rd | Easton, PA 18042
Office: 112 Skillman Library
p: (610) 330-5632
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210303/2085cba1/attachment.htm>