[slurm-users] Backfill Scheduling

[Brian Andrus]

Reed,

You may want to look at the timelimit aspect of the job(s).

For one to 'squeeze in', it needs to be able to finish before the 
resources in use are expected to become available.

Consider:
Job A is running on 2 nodes of a 3 node cluster. It will finish in 1 hour.
Pending job B will run for 2 hours needs 2 nodes, but only 1 is free, it 
waits.
Pending job C (with a lower priority) needs 1 node for 2 hours. Hmm, 
well it won't finish before the time job B is expected to start, so it 
waits.
Pending job D (with even lower priority) needs 1 node for 30 minutes. 
That can squeeze in before the additional node for Job B is expected to 
be available, so it runs on the idle node.

Brian Andrus

On 6/26/2023 3:48 PM, Reed Dier wrote:
> Hoping this will be an easy one for the community.
>
> The priority schema was recently reworked for our cluster, with only PriorityWeightQOS and PriorityWeightAge contributing to the priority value, while PriorityWeightAssoc, PriorityWeightFairshare, PriorityWeightJobSize, and PriorityWeightPartition are now set to 0, and PriorityFavorSmall set to NO.
> The cluster is fairly loaded right now, with a big backlog of work (~250 running jobs, ~40K pending jobs).
> The majority of these jobs are arrays, which runs the pending job count up quickly.
>
> What I’m trying to figure out is:
> The next highest priority job array in the queue is waiting on resources, everything else on priority, which makes sense.
> However, there is a good portion of the cluster unused, seemingly dammed by the next up job being large, while there are much smaller jobs behind it that could easily fit into the available resources footprint.
>
> Is this an issue with the relative FIFO nature of the priority scheduling currently with all of the other factors disabled,
> or since my queue is fairly deep, is this due to bf_max_job_test being the default 100, and it can’t look deep enough into the queue to find a job that will fit into what is unoccupied?
> PriorityType=priority/multifactor
> SchedulerType=sched/backfill
>
> Hoping to know where I might want to swing my hammer next, without whacking the wrong setting
>
> Appreciate any advice,
> Reed