[slurm-users] Slurp for sw builds
Cook, Malcolm
MEC at stowers.org
Thu Jan 4 19:17:11 UTC 2024
Alternatively, if your build process happens to already be implemented using GNU Make, it is possible to declare that (selected) recipes should not be executed using Gnu Makes default .SHELL (which is sh) but rather submitted to your cluster using slurm's `srun`. Is it?
> ~ mailto:malcolm_cook at stowers.org
>
> From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Renfro, Michael
> Sent: Wednesday, January 3, 2024 3:06 PM
> To: Slurm User Community List <slurm-users at lists.schedmd.com>
> Subject: Re: [slurm-users] Slurp for sw builds
>
> You can attack this in a few different stages. A lot of what you're interested in will be found at various university or national lab sites (I Googled "sbatch example" for the one below)
>
> 1. If you're good with doing a "make -j" to parallelize a make compilation over multiple CPUs in a single computer to build one executable, you can adapt the first example from the Multi-Threaded SMP Job section of [1], using the same number of cpus-per-task as you use for your -j flag.
> 2. Once that's going, you can use a script similar to the one at [2] to submit an array of concurrent build jobs, and they can spread out across multiple computers as needed. The built-in job array support will only change an environment variable SLURM_ARRAY_TASK_ID for each job in the array, but you can use that value to select a folder to cd into, or to do other things.
> 3. For a higher-level abstraction that will submit a bunch of build jobs, wait for them all to finish, and then archive the resulting artifacts, a pipeline tool like Snakemake [3] can track the individual tasks, and work with batch jobs or with local programs on the submission host. Two Slurm-related profiles for Snakemake are at [4] (simpler) and [5] (more comprehensive).
>
> [1] https://help.rc.ufl.edu/doc/Multi-Threaded_%26_Message_Passing_Job_Scripts
> [2] https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts#Array_job
> [3] https://snakemake.readthedocs.io/en/stable/
> [4] https://github.com/jdblischak/smk-simple-slurm
> [5] https://github.com/Snakemake-Profiles/slurm
>
> From: slurm-users <mailto:slurm-users-bounces at lists.schedmd.com> on behalf of Duane Ellis <mailto:duane at duaneellis.com>
> > Date: Wednesday, January 3, 2024 at 2:41 PM
> > To: mailto:slurm-users at lists.schedmd.com <mailto:slurm-users at lists.schedmd.com>
> > Subject: [slurm-users] Slurp for sw builds
> >
> > In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images
> >
> > My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster
> >
> > example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory
> >
> > The last step after the for loop would be wait for all of the child builds to complete
> >
> > Once all child jobs are done we have a single job that combines or packages all the intermediate images
> >
> > Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us)
> >
> > My question is this:
> >
> > Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install
> >
> > I see the srun and sbatch command man pages but no good examples
> >
> > Bonus would be something that integrates into a gitlab runner example or Jenkins in some way
> >
> > All I can seem to find is how to install and administer slurm not a how to use slurm
> >
> >
> > Sent from my iPhone
> >
>
More information about the slurm-users
mailing list