[slurm-users] Jobs waiting while plenty of cpu and memory available

[Thomas M. Payerle]

You can use squeue to see the priority of jobs.  I believe it normally
shows jobs in order of priority, even though does not display priority.  If
you want to see actual priority, you need to request it in the format
field.  I typically use
squeue -o "%.18i %.12a %.6P %.8u %.2t %.8m %.4D %.4C %12l %12p %Q %b %R"
<any other squeue options>

Do you have backfill enabled?  This can help in many cases.
If the job with highest priority is quite wide, Slurm will reserve
resources for it.  E.g., if it requests all of your nodes, then Slurm will
reserve all nodes as they become idle for the wide job, until no other jobs
are running and it can finally run.  W/out backfill, no other jobs will run
before it.  With backfill, Slurm will estimate when all the nodes needed
for the highest priority job to run will be available (based on walltime
limits of running jobs), and will allow other jobs to run on the reserved
nodes (backfill) as long as they will complete (based on their walltime
limits) before Slurm expects the remaining nodes for the top priority job
will be available.  This can greatly improve utilization of the cluster ---
I suspect a large percentage of our jobs run as backfill.


On Tue, Jul 9, 2019 at 10:10 AM Edward Ned Harvey (slurm) <
slurm at nedharvey.com> wrote:

> > From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of
> > Ole Holm Nielsen
> > Sent: Tuesday, July 9, 2019 2:36 AM
> >
> > When some jobs are pending with Reason=Priority this means that other
> > jobs with a higher priority are waiting for the same resources (CPUs) to
> > become available, and they will have Pending=Resources in the squeue
> > output.
>
> Yeah, that's exactly the problem. There are plenty of cpu and memory
> resources available, yet jobs are waiting. Is there any way to know what
> resources, specifically, the jobs are waiting for, or what jobs are ahead
> of a particular job in queue, so I can then look at what resources the
> first job requires? "scontrol show partition" doesn't reveal any clear
> problems:
>
>     PartitionName=batch
>        AllowGroups=ALL AllowAccounts=ALL DenyQos=foo,bar,baz
>        AllocNodes=ALL Default=YES QoS=N/A
>        DefaultTime=00:15:00 DisableRootJobs=NO ExclusiveUser=NO
> GraceTime=0 Hidden=NO
>        MaxNodes=UNLIMITED MaxTime=3-00:00:00 MinNodes=1 LLN=NO
> MaxCPUsPerNode=UNLIMITED
>        Nodes=alpha[003-068],omega[003-068]
>        PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
> OverSubscribe=NO
>        OverTimeLimit=NONE PreemptMode=REQUEUE
>        State=UP TotalCPUs=4321 TotalNodes=123 SelectTypeParameters=NONE
>        DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
>
> The QoS policies are not new, and have not changed recently, yet the
> problem of jobs pending is a new problem. I can't seem to get any
> information about why they're pending.
>
>
>

-- 
Tom Payerle
DIT-ACIGS/Mid-Atlantic Crossroads        payerle at umd.edu
5825 University Research Park               (301) 405-6135
University of Maryland
College Park, MD 20740-3831
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20190709/1a27e1f9/attachment.htm>