<html style="direction: ltr;">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body bidimailui-charset-is-forced="true" style="direction: ltr;"
text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 18/10/2018 20:34, Eli V wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJtFHURzj4LkAozBLwPYGshbiuBKY=V_y=_=wzFsCHDHKJrKWA@mail.gmail.com">
<pre wrap="">On Thu, Oct 18, 2018 at 1:03 PM Daniel Letai <a class="moz-txt-link-rfc2396E" href="mailto:dani@letai.org.il"><dani@letai.org.il></a> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
Hello all,
To solve a requirement where a large number of job arrays (~10k arrays, each with at most 8M elements) with same priority should be executed with minimal starvation of any array - we don't want to wait for each array to complete before starting the next one - we wish to implement "interleaving" between arrays, we came up with the following scheme:
Start all arrays in this partition in a "Hold" state.
Release a predefined number of elements (E.g., 200)
from this point a slurmctld prolog takes over:
On the 200th job run squeue, note the next job array (array id following the currently executing array id)
Release a predefined number of elements (E.g., 200)
and repeat
This might produce a very large number of release requests to the scheduler in a short time frame, and one concern is the scheduler loop getting too many requests.
Can you think of other issues that might come up with this approach?
Do you have any recommendations, or might suggest a better approach to solve this problem?
</pre>
</blockquote>
<pre wrap="">
I can't comment on the scalability issues but if possible using %200
on the array submission seems like the simplest solution. From the
sbatch man page:
For example "--array=0-15%4" will limit the number of simultaneously
running tasks from this job array to 4.</pre>
</blockquote>
Won't achieve the same solution - we want a round-robin solution,
your proposal would hard limit to 200 jobs per array. This will
leave most of the cluster underutilized.<br>
<blockquote type="cite"
cite="mid:CAJtFHURzj4LkAozBLwPYGshbiuBKY=V_y=_=wzFsCHDHKJrKWA@mail.gmail.com">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">
We have considered fairshare, but all arrays are from same account and user. We have considered creating accounts on the fly (1 for each array) but get an error ("This should never happen") after creating a few thousand accounts.
To my understanding fairshare is only viable between accounts.
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Regards,
Daniel Letai
+972 (0)505 870 456</pre>
</body>
</html>