[slurm-users] job stuck as pending - reason "PartitionConfig"

[byron]

Hi

When I try to submit a job to one of our partitions it just stay in the
stay pending with the reason "PartitionConfig".  Can someone point me in
the right direction for how to troubleshoot this?  I'm a bit stumpped.

Some details of the setup

The version is 19.05.7

This is the job that is stuck in state pending
             JOBID PARTITION     NAME     USER ST       TIME  NODES
NODELIST(REASON)
          10860160   highmem MooseBen byron PD       0:00     16
(PartitionConfig)

$ sinfo -p highmem
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
highmem      up   infinite      1  drain intel-0012
highmem      up   infinite     19   idle intel-[0001-0011,0013-0020]

The output from  scontrol show part
PartitionName=highmem
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=02:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
MaxCPUsPerNode=UNLIMITED
   Nodes=intel-00[01-20]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=EXCLUSIVE
   OverTimeLimit=NONE PreemptMode=REQUEUE
   State=UP TotalCPUs=320 TotalNodes=20 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210929/f192a382/attachment.htm>