<div dir="ltr"><div><br></div>HI,<div><br></div><div>Short answer: scontrol update jobid=JOBID arraytaskthrottle=NEWLIMIT</div><div><br></div><div>Long answer: <a href="https://bugs.schedmd.com/show_bug.cgi?id=1863">https://bugs.schedmd.com/show_bug.cgi?id=1863</a></div><div class="gmail_extra"><br></div><div class="gmail_extra">jbh</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 24, 2018 at 5:55 AM, Bill Barth <span dir="ltr"><<a href="mailto:bbarth@tacc.utexas.edu" target="_blank">bbarth@tacc.utexas.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We don’t allow array jobs (we have our own tools for packing small jobs into bigger ones), so I can’t look at this myself, but what does ‘scontrol show job <jobid>’ show for this job? If you can find the ‘4’ in this job as some named parameter, you ought to be able to do an ‘scontrol update job <jobid> ThatParameter=100’ or whatever you like to change it.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Bill Barth, Ph.D., Director, HPC<br>
<a href="mailto:bbarth@tacc.utexas.edu">bbarth@tacc.utexas.edu</a> | Phone: <a href="tel:%28512%29%20232-7069" value="+15122327069">(512) 232-7069</a><br>
Office: ROC 1.435 | Fax: <a href="tel:%28512%29%20475-9445" value="+15124759445">(512) 475-9445</a><br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On 2/23/18, 11:13 PM, "slurm-users on behalf of ~Stack~" <<a href="mailto:slurm-users-bounces@lists.schedmd.com">slurm-users-bounces@lists.<wbr>schedmd.com</a> on behalf of <a href="mailto:i.am.stack@gmail.com">i.am.stack@gmail.com</a>> wrote:<br>
<br>
Greetings,<br>
<br>
I have a user that submits many many many jobs at once in an array.<br>
Happily, he's a very nice user and doesn't often cause trouble.<br>
<br>
The documentation for the job array<br>
(<a href="https://slurm.schedmd.com/job_array.html" rel="noreferrer" target="_blank">https://slurm.schedmd.com/<wbr>job_array.html</a>) says:<br>
"A maximum number of simultaneously running tasks from the job array may<br>
be specified using a "%" separator. For example "--array=0-15%4" will<br>
limit the number of simultaneously running tasks from this job array to 4."<br>
<br>
Awesome. That's exactly what he is doing.<br>
<br>
A big job recently just finished and I'm looking at my queue noticing<br>
that no one bothered to load it to the brim this weekend leaving me with<br>
several idle compute nodes. Meanwhile, this user has got quite a few<br>
jobs still waiting to run with a "JobArrayTaskLimit". :-/<br>
<br>
I've been poking at it for the last 20-30 minutes, but I'm not seeing<br>
how I, with the power of root, can update his own "self imposed" array<br>
limit. It's late, my attempts have not worked, and my google-fu isn't<br>
returning any helpful results. I'm not really concerned by it, but I<br>
would like to know should this happen again.<br>
<br>
How can I increase a JobArrayTaskLimit?<br>
So using the documentation example, how would I "scontrol update" the<br>
array to be "--array=0-15%6" when jobs 0-3 are already running?<br>
<br>
Or maybe just say "Grab X number of jobs and run them anyway"?<br>
So again with the documentation example, maybe 0-3 are done, 4-7 are<br>
running, and I just want to manually tell 8-10 to run anyway on<br>
available resources leaving 11-15 under current constraints.<br>
<br>
Thank you!<br>
~Stack~<br>
<br>
<br>
<br>
</div></div></blockquote></div><br></div></div>