[slurm-users] maximum size of array jobs

[Marcus Wagner]

Hi Merlin,

thanks for the answer, but our user is not in need of a high index, but 
in fact in need of 100k taskids.


Best
Marcus


On 2/26/19 3:50 PM, Merlin Hartley wrote:
> *max_array_tasks*
>     Specify the maximum number of tasks that be included in a job
>     array. The default limit is MaxArraySize, but this option can be
>     used to set a lower limit. For example, max_array_tasks=1000 and
>     MaxArraySize=100001 would permit a maximum task ID of 100000, but
>     limit the number of tasks in any single job array to 1000.
> https://slurm.schedmd.com/slurm.conf.html
>
> SchedulerParameters=max_array_tasks=1000
>
> MaxArraySize=100000
>
> See commit:
> https://github.com/SchedMD/slurm/commit/09c13fb292a4a6a56b4078de840aae0d4db70309 
>
>
>
>
> --
> Merlin Hartley
> Computer Officer
> MRC Mitochondrial Biology Unit
> University of Cambridge
> Cambridge, CB2 0XY
> United Kingdom
>
>> On 26 Feb 2019, at 14:27, Jeffrey Frey <frey at udel.edu 
>> <mailto:frey at udel.edu>> wrote:
>>
>> Also see "https://slurm.schedmd.com/slurm.conf.html" for 
>> MaxArraySize/MaxJobCount.
>>
>> We just went through a user-requested adjustment to MaxArraySize to 
>> bump it from 1000 to 10000; as the documentation states, since each 
>> index of an array job is essentially "a job," you must be sure to 
>> also adjust MaxJobCount (from 10000 to 100000 in our case). 
>>  Adjusting MaxJobCount requires a restart of slurmctld; though the 
>> documentation doesn't state it, so does adjustment of MaxArraySize 
>> (scontrol reconfigure will succeed but leave the previous limit in 
>> effect, see "https://bugs.schedmd.com/show_bug.cgi?id=6553").
>>
>> The "MaxArraySize" is a bit of a misnomer since it's really 1 + the 
>> top of the valid range of indices -- "MaxArrayIndex" would be more 
>> apt.  Our users were very happy with Grid Engine's allowance of any 
>> index range and striding that produces no more than "max_aj_tasks" 
>> indices; since moving to Slurm they're forced to come up with their 
>> own index-mapping functionality at times, but the relatively low 
>> MaxArraySize versus what we had in GridEngine (75000) has been 
>> especially frustrating for them.
>>
>> So far the 10000/100000 combo hasn't come close to exhausting 
>> resources on our slurmctld nodes; but we haven't actually submitted a 
>> couple 10000-index array jobs and enough other jobs to hit 100000 
>> active jobs, so current memory usage isn't an adequate measure of 
>> usage under load.  Since the slurm.conf documentation states:
>>
>>
>>     Performance can suffer with more than a few hundred thousand jobs.
>>
>>
>>
>> we're reluctant to increase MaxJobCount too much higher.
>>
>>
>>
>>
>>> On Feb 26, 2019, at 3:18 AM, Ole Holm Nielsen 
>>> <Ole.H.Nielsen at fysik.dtu.dk <mailto:Ole.H.Nielsen at fysik.dtu.dk>> wrote:
>>>
>>> On 2/26/19 9:07 AM, Marcus Wagner wrote:
>>>> Does anyone know, why per default the number of array elements is 
>>>> limited to 1000?
>>>> We have one user, who would like to have 100k array elements!
>>>> What is more difficult for the scheduler, one array job with 100k 
>>>> elements or 100k non-array jobs?
>>>> Where did you set the limit? Do your users use array jobs at all?
>>>
>>> Google is your friend :-)
>>>
>>> https://slurm.schedmd.com/job_array.html
>>>
>>>> A new configuration parameter has been added to control the maximum 
>>>> job array size: MaxArraySize. The smallest index that can be 
>>>> specified by a user is zero and the maximum index is MaxArraySize 
>>>> minus one. The default value of MaxArraySize is 1001. The maximum 
>>>> MaxArraySize supported in Slurm is 4000001. Be mindful about the 
>>>> value of MaxArraySize as job arrays offer an easy way for users to 
>>>> submit large numbers of jobs very quickly.
>>>
>>> /Ole
>>>
>>
>>
>> ::::::::::::::::::::::::::::::::::::::::::::::::::::::
>> Jeffrey T. Frey, Ph.D.
>> Systems Programmer V / HPC Management
>> Network & Systems Services / College of Engineering
>> University of Delaware, Newark DE  19716
>> Office: (302) 831-6034  Mobile: (302) 419-4976
>> ::::::::::::::::::::::::::::::::::::::::::::::::::::::
>>
>>
>>
>>
>

-- 
Marcus Wagner, Dipl.-Inf.

IT Center
Abteilung: Systeme und Betrieb
RWTH Aachen University
Seffenter Weg 23
52074 Aachen
Tel: +49 241 80-24383
Fax: +49 241 80-624383
wagner at itc.rwth-aachen.de
www.itc.rwth-aachen.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190226/780723e9/attachment.html>