In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images
My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster
example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory
The last step after the for loop would be wait for all of the child builds to complete
Once all child jobs are done we have a single job that combines or packages all the intermediate images
Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us)
My question is this:
Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install
I see the srun and sbatch command man pages but no good examples
Bonus would be something that integrates into a gitlab runner example or Jenkins in some way
All I can seem to find is how to install and administer slurm not a how to use slurm
Sent from my iPhone
You can attack this in a few different stages. A lot of what you’re interested in will be found at various university or national lab sites (I Googled “sbatch example” for the one below)
1. If you’re good with doing a “make -j” to parallelize a make compilation over multiple CPUs in a single computer to build one executable, you can adapt the first example from the Multi-Threaded SMP Job section of [1], using the same number of cpus-per-task as you use for your -j flag. 2. Once that’s going, you can use a script similar to the one at [2] to submit an array of concurrent build jobs, and they can spread out across multiple computers as needed. The built-in job array support will only change an environment variable SLURM_ARRAY_TASK_ID for each job in the array, but you can use that value to select a folder to cd into, or to do other things. 3. For a higher-level abstraction that will submit a bunch of build jobs, wait for them all to finish, and then archive the resulting artifacts, a pipeline tool like Snakemake [3] can track the individual tasks, and work with batch jobs or with local programs on the submission host. Two Slurm-related profiles for Snakemake are at [4] (simpler) and [5] (more comprehensive).
[1] https://help.rc.ufl.edu/doc/Multi-Threaded_%26_Message_Passing_Job_Scripts [2] https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts#Array_job [3] https://snakemake.readthedocs.io/en/stable/ [4] https://github.com/jdblischak/smk-simple-slurm [5] https://github.com/Snakemake-Profiles/slurm
From: slurm-users slurm-users-bounces@lists.schedmd.com on behalf of Duane Ellis duane@duaneellis.com Date: Wednesday, January 3, 2024 at 2:41 PM To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Slurp for sw builds External Email Warning
This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.
________________________________
In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images
My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster
example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory
The last step after the for loop would be wait for all of the child builds to complete
Once all child jobs are done we have a single job that combines or packages all the intermediate images
Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us)
My question is this:
Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install
I see the srun and sbatch command man pages but no good examples
Bonus would be something that integrates into a gitlab runner example or Jenkins in some way
All I can seem to find is how to install and administer slurm not a how to use slurm
Sent from my iPhone
Alternatively, if your build process happens to already be implemented using GNU Make, it is possible to declare that (selected) recipes should not be executed using Gnu Makes default .SHELL (which is sh) but rather submitted to your cluster using slurm's `srun`. Is it?
~ mailto:malcolm_cook@stowers.org
From: slurm-users slurm-users-bounces@lists.schedmd.com On Behalf Of Renfro, Michael Sent: Wednesday, January 3, 2024 3:06 PM To: Slurm User Community List slurm-users@lists.schedmd.com Subject: Re: [slurm-users] Slurp for sw builds
You can attack this in a few different stages. A lot of what you're interested in will be found at various university or national lab sites (I Googled "sbatch example" for the one below)
- If you're good with doing a "make -j" to parallelize a make compilation over multiple CPUs in a single computer to build one executable, you can adapt the first example from the Multi-Threaded SMP Job section of [1], using the same number of cpus-per-task as you use for your -j flag.
- Once that's going, you can use a script similar to the one at [2] to submit an array of concurrent build jobs, and they can spread out across multiple computers as needed. The built-in job array support will only change an environment variable SLURM_ARRAY_TASK_ID for each job in the array, but you can use that value to select a folder to cd into, or to do other things.
- For a higher-level abstraction that will submit a bunch of build jobs, wait for them all to finish, and then archive the resulting artifacts, a pipeline tool like Snakemake [3] can track the individual tasks, and work with batch jobs or with local programs on the submission host. Two Slurm-related profiles for Snakemake are at [4] (simpler) and [5] (more comprehensive).
[1] https://help.rc.ufl.edu/doc/Multi-Threaded_%26_Message_Passing_Job_Scripts [2] https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts#Array_job [3] https://snakemake.readthedocs.io/en/stable/ [4] https://github.com/jdblischak/smk-simple-slurm [5] https://github.com/Snakemake-Profiles/slurm
From: slurm-users mailto:slurm-users-bounces@lists.schedmd.com on behalf of Duane Ellis mailto:duane@duaneellis.com
Date: Wednesday, January 3, 2024 at 2:41 PM To: mailto:slurm-users@lists.schedmd.com mailto:slurm-users@lists.schedmd.com Subject: [slurm-users] Slurp for sw builds
In my case I would like to use a slurm cluster for a sw ci/cd like solution for building sw images
My current scripted full system build takes 3-5 hours and is done serially we could easily find places where we can choose to build things in parallel hence the idea is to spawn parallel builds on the Linux slurm cluster
example: we have a list of images we iterate over in a for loop to build each thing the steps are: cd somedir then type make or run a shell script in that directory
The last step after the for loop would be wait for all of the child builds to complete
Once all child jobs are done we have a single job that combines or packages all the intermediate images
Really want to use slurm because our FPGA team will have a giant slurm linux cluster for Xilinix FPGA builds and those nodes can do what we need for sw purposes (reusing the existing cluster is a huge win for us)
My question is this:
Can somebody point me to some sw build examples for or using slurm? All I can seem to find is how to install
I see the srun and sbatch command man pages but no good examples
Bonus would be something that integrates into a gitlab runner example or Jenkins in some way
All I can seem to find is how to install and administer slurm not a how to use slurm
Sent from my iPhone
-
Cook, Malcolm -
Duane Ellis -
Renfro, Michael