[slurm-users] requesting resources and afterwards launch an array of calculations
Alfredo Quevedo
maquevedo.unc at gmail.com
Tue Dec 18 12:58:45 MST 2018
Thank you Aaron for the reply,
Specifically I am trying to run what in chemistry is known as an
Umbrella Samplig simulation, in which independent simulation windows are
run. The total number of windows for the whole simulation is 104, but
allocating 104 cores to perform the simulation would signify a long wait
time in the queue. So, sometime ago I found a solution to this in the
execution of an array, which will take profit of idle cores and that
will run the jos sequentially. That is launch n array jobs (lets say 30)
of the total number of jobs (i.e. 104) and after each job finishes the
new window is started in case the resource is available. All the
simulation windows are independent from each other and runs for more or
less the same time. My problem is that if the maximum number of jobs in
the queue is set to a low value, the array is launched to reach the
maxjobslimit although a higher number of cores are free. Sorry if I am
not so clear, but I am not a computer scientist :-)
El 19/12/2018 a las 12:25, Aaron Knister escribió:
> Alfredo,
>
> I’m assuming the resources are used initially in some sort of tightly-coupled parallel task, or at least some workload where all the tasks finish at about the same time. I’m wondering and also assuming that the tasks you’re looking to run afterwards as part of an array are less tightly coupled. Does this mean there might be significant variation in the runtime of those tasks? I’m curious if your desire here is to have those array jobs treated as individual allocations so that resources can free up as tasks complete (e.g so that you’re not holding up 30 processors if just one of the tasks takes significantly longer to complete than the others).
>
> Sent from my iPhone
>
>> On Dec 18, 2018, at 12:03, Alfredo Quevedo <maquevedo.unc at gmail.com> wrote:
>>
>> Dear slurm users,
>>
>> I would like to know if it is possible to prepare a slurm submission script in a way that initially CPU resources are requested (lets say 30 CPUs), and afterwards, the assigned resources are used to launch an array of 30 single CPU jobs array? I would greatly appreciate any hint in this respect
>>
>> Thanks in advance
>>
>> Alfredo
>>
>>
More information about the slurm-users
mailing list