[slurm-users] ticking time bomb? launching too many jobs in parallel

Fri Aug 30 06:33:55 UTC 2019

Hi Steven,

Thanks for taking the time to reply to my post.

Setting a limit on the number of jobs for a single array isn't sufficient
because regression-tests need to launch multiple arrays, and I would need a
job limit that would take effect over all launched jobs.

It's very possible I'm not understand something. I'll lay out a very
specific example in the hopes you can correct me if I've gone wrong
somewhere.

Let's take the small cluster with 140 GPUs and no fairshare as an example,
because it's easier for me to explain.

The users, who all know each other personally and interact via chat, decide
on a daily basis how many jobs each user can run at a time.

Let's say today is Sunday (hypothetically). Nobody is actively developing
today, except that user 1 has 10 jobs running for the entire weekend. That
leaves 130 GPUs unused.

User 2, whose jobs all run on 1 GPU decides to run a regression test. The
regression test comprises of 9 different scripts each run 40 times, for a
grand total of 360 jobs. The duration of the scripts vary from 1 and 5
hours to complete, and the jobs take on average 4 hours to complete.

User 2 gets the user group's approval (via chat) to use 90 GPUs (so that 40
GPUs will remain for anyone else wanting to work that day).

The problem I'm trying to solve is this: how do I ensure that user 2
launches his 360 jobs in such a way that 90 jobs are in the run state
consistently until the regression test is finished?

Keep in mind that:

   - limiting each job array to 10 jobs is inefficient: when the first job
   array finishes (long before the last one), only 80 GPUs will be used, and
   so on as other arrays finish
   - the admin is not available, he cannot be asked to set a hard limit of
   90 jobs for user 2 just for today

I would be happy to use job arrays if they allow me to set an overarching
job limit across multiple arrays. Perhaps this is doable. Admttedly I'm
working on a paper to be submitted in a few days, so I don't have time to
test jobs arrays thoroughly, but I will try out job arrays more thoroughly
once I've submitted my paper (ie after sept 5).

My solution, for now, is to not use job arrays. Instead, I launch each job
individually, and I use singleton (by launching all jobs with the same 90
unique names) to ensure that exactly 90 jobs are run at a time (in this
case, corresponding to 90 GPUs in use).

Side note: the unavailability of the admin might sound contrived by picking
Sunday as an example, but it's in fact very typical. The admin is not
available:

   - on weekends (the present example)
   - at any time outside of 9am to 5pm (keep in mind, this is a cluster
   used by students in different time zones)
   - any time he is on vacation
   - anytime the he is looking after his many other responsibilities.
   Constantly setting user limits that change on a daily basis would be too
   much too ask.

I'd be happy if you corrected my misunderstandings, especially if you could
show me how to set a job limit that takes effect over multiple job arrays.

I may have very glaring oversights as I don't necessarily have a big
picture view of things (I've never been an admin, most notably), so feel
free to poke holes at the way I've constructed things.

Regards,
Guillaume.

On Fri, Aug 30, 2019 at 1:22 AM Steven Dick <kg4ydw at gmail.com> wrote:

> This makes no sense and seems backwards to me.
>
> When you submit an array job, you can specify how many jobs from the
> array you want to run at once.
> So, an administrator can create a QOS that explicitly limits the user.
> However, you keep saying that they probably won't modify the system
> for just you...
>
> That seems to me to be the perfect case to use array jobs and tell it
> how many elements of the array to run at once.
> You're not using array jobs for exactly the wrong reason.
>
> On Tue, Aug 27, 2019 at 1:19 PM Guillaume Perrault Archambault
> <gperr050 at uottawa.ca> wrote:
> > The reason I don't use job arrays is to be able limit the number of jobs
> per users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190830/8a9a356a/attachment-0001.htm>