[slurm-users] ticking time bomb? launching too many jobs in parallel

[Steven Dick]

On Fri, Aug 30, 2019 at 2:58 PM Guillaume Perrault Archambault
<gperr050 at uottawa.ca> wrote:
> My problem with that though, is what if each script (the 9 scripts in my earlier example) each require different requirements? For example, run on a different partition, or set a different time limit? My understanding is that for a single job array, each job will get the same job requirements.

That's a little messier and may be less suitable for an array job.
However, some of that can be accomplished.   You can for instance,
submit a job to multiple partitions and then use srun within the job
to allocate resources to individual tasks within the job.
But you get a lot less control over how the resources are spread, so
it might not be workable.

> The other problem is that with the way I've implemented it, I can change the max jobs dynamically.

Others have indicated in this thread that qos can be dynamically
changed; I don't recall trying that, but if you did, I think you'd do
it with scontrol.