[slurm-users] Job array start time and SchedNodes

[Thekla Loizou]

Dear Loris,

Thank you for your reply. I don't believe that there is something wrong 
with the job configuration or the node configuration to be honest.

I have just submitted a simple sleep script:

#!/bin/bash

sleep 10

as below:

sbatch --array=1-10 --ntasks-per-node=40 --time=09:00:00 test.sh

and squeue shows:

           131799_1       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_2       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_3       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_4       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_5       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_6       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_7       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_8       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
           131799_9       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)
          131799_10       cpu  test.sh   thekla PD N/A      1 
cn04                 (Priority)

All of the jobs seem to be scheduled on node cn04.

When they start running they run on separate nodes:

           131799_1       cpu  test.sh   thekla  R       0:02 1 cn01
           131799_2       cpu  test.sh   thekla  R       0:02 1 cn02
           131799_3       cpu  test.sh   thekla  R       0:02 1 cn03
           131799_4       cpu  test.sh   thekla  R       0:02 1 cn04

Regards,

Thekla

On 7/12/21 5:17 μ.μ., Loris Bennett wrote:
> Dear Thekla,
>
> Thekla Loizou <t.loizou at cyi.ac.cy> writes:
>
>> Dear Loris,
>>
>> There is no specific node required for this array. I can verify that from
>> "scontrol show job 124841" since the requested node list is empty:
>> ReqNodeList=(null)
>>
>> Also, all 17 nodes of the cluster are identical so all nodes fulfill the job
>> requirements, not only node cn06.
>>
>> By "saving" the other nodes I mean that the scheduler estimates that the array
>> jobs will start on 2021-12-11T03:58:00. No other jobs are scheduled to run
>> during that time on the other nodes. So it seems that somehow the scheduler
>> schedules the array jobs on more than one nodes but this is not showing in the
>> squeue or scontrol output.
> My guess is that there is something wrong with either the job
> configuration or the node configuration, if Slurm thinks 9 jobs which
> require a whole node can all be started simultaneously on same node.
>
> Cheers,
>
> Loris
>
>> Regards,
>>
>> Thekla
>>
>>
>> On 7/12/21 12:16 μ.μ., Loris Bennett wrote:
>>> Hi Thekla,
>>>
>>> Thekla Loizou <t.loizou at cyi.ac.cy> writes:
>>>
>>>> Dear all,
>>>>
>>>> I have noticed that SLURM schedules several jobs from a job array on the same
>>>> node with the same start time and end time.
>>>>
>>>> Each of these jobs requires the full node. You can see the squeue output below:
>>>>
>>>>             JOBID     PARTITION  ST   START_TIME          NODES SCHEDNODES
>>>> NODELIST(REASON)
>>>>
>>>>             124841_1       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_2       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_3       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_4       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_5       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_6       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_7       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_8       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>             124841_9       cpu     PD 2021-12-11T03:58:00      1
>>>> cn06                 (Priority)
>>>>
>>>> Is this a bug or am I missing something? Is this because the jobs have the same
>>>> JOBID and are still in pending state? I am aware that the jobs will not actually
>>>> all run on the same node at the same time and that the scheduler somehow takes
>>>> into account that this job array has 9 jobs that will need 9 nodes. I am
>>>> creating a timeline with the start time of all jobs and when the job array jobs
>>>> will start running no other jobs are set to run on the remaining nodes (so it
>>>> "saves" the other nodes for the jobs of the array even if they are all scheduled
>>>> to run on the same node based on squeue or scontrol).
>>> In general jobs from an array will be scheduled on whatever nodes
>>> fulfil their requirements.  The fact that all the jobs have
>>>
>>>     cn06
>>>
>>> as NODELIST however seems to suggest that you have either specified cn06
>>> as the node the jobs should run on, or cn06 is the only node which
>>> fulfils the job requirements.
>>>
>>> I'm not sure what you mean about '"saving" the other nodes'.
>>>
>>> Cheers,
>>>
>>> Loris
>>>