[slurm-users] [EXT] Weird issues with slurm's Priority

[Sean Crosby]

Hi,

What you have described is how the backfill scheduler works. If a lower
priority job can start AND finish before the resources a higher priority
job requires are available, the backfill scheduler will start the lower
priority job.

Your high priority job requires 24 cores, whereas the lower priority jobs
only require 1 core each. Therefore there might be some free resources the
lower priority jobs can use that the 24 core job can't. The backfill
scheduler can make the lower priority jobs take advantage of those free
cores, but only if it then doesn't stop the higher priority job from
starting in its original time.

In your example job list, can you also list the requested times for each
job? That will show if it is the backfill scheduler doing what it is
designed to do.

Sean

--
Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead
Research Computing Services | Business Services
The University of Melbourne, Victoria 3010 Australia



On Tue, 7 Jul 2020 at 19:05, zaxs84 <sciuscianebbia at gmail.com> wrote:

> *UoM notice: External email. Be cautious of links, attachments, or
> impersonation attempts.*
> ------------------------------
> Hi all.
>
> We want to achieve a simple thing with slurm: launch "normal" jobs, and be
> able to launch "high priority" jobs that run as soon as possible. End of
> it. However we cannot achieve this in a reliable way, meaning that our
> current config sometimes works, sometimes not, and this is driving us crazy.
>
> When it works, this is what happens:
> - we have, let's say, 10 jobs running with normal priority (--qos=normal,
> having final Priority=1001) and few thousands in PENDING state
> - we submit a new job with high priority (--qos=high, having final
> Priority=1001001)
> - at this point, slurm waits until the normal priority job will end to
> free up required resources, and then starts a new High priority job. That's
> Perfect!
>
> However, from time to time, randomly, this does not happen. Here is an
> example:
>
> # the node has around 200GB of memory and 24 CPUs
> Partition=t1 State=PD Priority=1001001 Nice=0 ID=337455 CPU=24 Memory=80G
> Nice=0 Started=0:00 User=u1 Submitted=2020-07-07T07:16:47
> Partition=t1 State=R Priority=1001 Nice=0 ID=337475 CPU=1 Memory=1024M
> Nice=0 Started=1:22 User=u1 Submitted=2020-07-07T10:31:46
> Partition=t1 State=R Priority=1001 Nice=0 ID=334355 CPU=1 Memory=1024M
> Nice=0 Started=58:09 User=u1 Submitted=2020-06-23T09:57:11
> Partition=t1 State=R Priority=1001 Nice=0 ID=334354 CPU=1 Memory=1024M
> Nice=0 Started=6:29:59 User=u1 Submitted=2020-06-23T09:57:11
> Partition=t1 State=R Priority=1001 Nice=0 ID=334353 CPU=1 Memory=1024M
> Nice=0 Started=13:25:55 User=u1 Submitted=2020-06-23T09:57:11
> [...]
>
> You see? Slurm keep starting jobs that have a lower priority. Why is that?
>
> Some info about our config: Slurm is version 16.05. Here is the priority
> config of slurm:
>
> ##### file /etc/slurm-llnl/slurm.conf
> PriorityType=priority/multifactor
> PriorityFavorSmall=NO
> PriorityWeightQOS=1000000
> PriorityWeightFairshare=1000
> PriorityWeightPartition=1000
> PriorityWeightJobSize=0
> PriorityWeightAge=0
>
> ##### command "sacctmgr show qos"
>       Name   Priority  MaxSubmitPA
>     normal          0  30
>       high       1000
>
>
> Any idea?
>
> Thanks
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20200707/6339fa82/attachment.htm>