[slurm-users] backfill scheduler does not work for heterogeneous jobs (version 17.11)

[Ana Jokanović]

Hi Ken,

I have read this page and I understood that in case of my example the third
job should be backfilled. The second job can start after 15 minutes, but
the third job requires only two nodes and 2 minutes, thus it can start
immediately, but this does not happen.

In the page that you referred to, they give an example:

For example, consider a heterogeneous job with three components. When
considered as independent jobs, the components could be initiated at times
now (component 0), now plus 2 hour (component 1), and now plus 1 hours
(component 2). When the backfill scheduler runs in the first mode:

   1. Component 0 will be noted to possible to start now, but not initiated
   due to the additional components to be initiated
   2. Component 1 will be noted to be possible to start in 2 hours
   3. Component 2 will not be considered for scheduling until 2 hours in
   the future,* which leave some additional resources available for
   scheduling to other jobs*

When the backfill scheduler executes next, it will use the second mode and
(assuming no other state changes) all three job components will be
considered available for scheduling no earlier than 2 hours in the
future, *which
may allow other jobs to be allocated resources before heterogeneous job
component 0 could be initiated.*
>From this example, I understand that in my experiment the third job should
be backfilled. The second job can start after 15 minutes, but the third job
requires only two nodes and 2 minutes, thus it can start immediately, but
this does not happen.

It seems there is a bug here. I also tried with the version 18.03, but it
does not work either.

Ana


On Fri, 30 Nov 2018 at 17:46, Kenneth Roberts <kroberts at materialsdesign.com>
wrote:

> There are some Limitations that mention backfill on the heterogeneous job
> support page.
>
>
>
> https://slurm.schedmd.com/heterogeneous_jobs.html#limitations
>
>
>
> Maybe there’s some information there to help?
>
>
>
> Ken
>
>
>
> *From:* slurm-users <slurm-users-bounces at lists.schedmd.com> *On Behalf Of
> *Ana Jokanovic
> *Sent:* Thursday, November 29, 2018 4:28 AM
> *To:* slurm-users at lists.schedmd.com
> *Subject:* [slurm-users] backfill scheduler does not work for
> heterogeneous jobs (version 17.11)
>
>
>
>
>
>
>
> Hello,
>
>
>
> I did a simple test submitting the workload of three jobs (see below) on a
> cluster of 5 nodes:
>
>
>
> sbatch --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2
> --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15
>
> sbatch --cpus-per-task=2 --ntasks=6 --time=15 : --cpus-per-task=2
> --ntasks=6 --time=15 : --cpus-per-task=2 --ntasks=6 --time=15
>
> sleep 5;
>
> sbatch --ntasks=1 --time=2 : --ntasks=1 --time=1
>
>
>
> I would expect that the third submitted job is backfilled but it does not
> happen.
>
> Here is the job completion log:
>
>
>
> JobId=2 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b09 NodeCnt=1
> ProcCnt=48
>
> JobId=3 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b10 NodeCnt=1
> ProcCnt=48
>
> JobId=4 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317714 EndTime=1543317774 NodeList=s19r2b12 NodeCnt=1
> ProcCnt=48
>
> JobId=8 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:02:00 SubmitTime=1543317699
> StartTime=1543317804 EndTime=1543317824 NodeList=s19r2b14 NodeCnt=1
> ProcCnt=48
>
> JobId=9 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:01:00 SubmitTime=1543317699
> StartTime=1543317804 EndTime=1543317824 NodeList=s19r2b16 NodeCnt=1
> ProcCnt=48
>
> JobId=5 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b09 NodeCnt=1
> ProcCnt=48
>
> JobId=6 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b10 NodeCnt=1
> ProcCnt=48
>
> JobId=7 UserId=3113 GroupId=8950 Name=sleep JobState=COMPLETED
> Partition=debug TimeLimit=00:15:00 SubmitTime=1543317694
> StartTime=1543317804 EndTime=1543317864 NodeList=s19r2b12 NodeCnt=1
> ProcCnt=48
>
>
>
> Would you expect this behavior?
>
>
>
> Thanks.
>
>
>
> Best regards,
>
> Ana
>
> --
>
> Ana Jokanovic, PhD
> Barcelona Supercomputing Center
> c/ Jordi Girona 1-3, K2M Building, 1st floor
> 08034 Barcelona - SPAIN
> e-mail: anaj82 at gmail.com or ana.jokanovic at bsc.es
> tel: +34 93 4137246
>
>
>
>
> --
>
> Ana Jokanovic, PhD
> Barcelona Supercomputing Center
> c/ Jordi Girona 1-3, K2M Building, 1st floor
> 08034 Barcelona - SPAIN
> e-mail: anaj82 at gmail.com or ana.jokanovic at bsc.es
> tel: +34 93 4137246
>
>
>
>
> --
>
> Ana Jokanovic, PhD
> Barcelona Supercomputing Center
> c/ Jordi Girona 1-3, K2M Building, 1st floor
> 08034 Barcelona - SPAIN
> e-mail: anaj82 at gmail.com or ana.jokanovic at bsc.es
> tel: +34 93 4137246
>


-- 
Ana Jokanovic, PhD
Barcelona Supercomputing Center
c/ Jordi Girona 1-3, K2M Building, 1st floor
08034 Barcelona - SPAIN
e-mail: anaj82 at gmail.com or ana.jokanovic at bsc.es
tel: +34 93 4137246
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181203/44020658/attachment-0001.html>