<div dir="ltr"><div>Bingo!</div><div><br></div><div>You were right, I was asking for more cores than was available (our highmem nodes have less than out standard nodes). I was so convinced that the problem was related to my upgrading the OS on those nodes that it never crossed my mind that it was something as straightforward as that.</div><div><br></div><div>Thanks for your help.<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 29, 2021 at 7:49 PM Paul Brunk <<a href="mailto:pbrunk@uga.edu">pbrunk@uga.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-GB">
<div class="gmail-m_1973921330213401468WordSection1">
<p class="MsoNormal"><span>Hello Byron:<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span>I’m guessing that your job is asking for more HW than the highmem_p<u></u><u></u></span></p>
<p class="MsoNormal"><span>has in it, or more cores or RAM within a node than any of the nodes<u></u><u></u></span></p>
<p class="MsoNormal"><span>have, or something like that. 'scontrol show job 10860160' might<u></u><u></u></span></p>
<p class="MsoNormal"><span>help. You can also look in slurmctld.log for that jobid.<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
<p class="MsoNormal">Paul Brunk, system administrator<u></u><u></u></p>
<p class="MsoNormal">Georgia Advanced Computing Resource Center<u></u><u></u></p>
<p class="MsoNormal">Enterprise IT Svcs, the University of Georgia<u></u><u></u></p>
</div>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" target="_blank">slurm-users-bounces@lists.schedmd.com</a>>
<b>On Behalf Of </b>byron<br>
<b>Sent:</b> Wednesday, September 29, 2021 10:35<br>
<b>To:</b> Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" target="_blank">slurm-users@lists.schedmd.com</a>><br>
<b>Subject:</b> [slurm-users] job stuck as pending - reason "PartitionConfig"<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(186,12,47)">[EXTERNAL SENDER - PROCEED CAUTIOUSLY]</span><u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Hi<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When I try to submit a job to one of our partitions it just stay in the stay pending with the reason "PartitionConfig". Can someone point me in the right direction for how to troubleshoot this? I'm a bit stumpped.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Some details of the setup<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The version is 19.05.7<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This is the job that is stuck in state pending<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)<br>
10860160 highmem MooseBen byron PD 0:00 16 (PartitionConfig)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">$ sinfo -p highmem<br>
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>
highmem up infinite 1 drain intel-0012<br>
highmem up infinite 19 idle intel-[0001-0011,0013-0020]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The output from scontrol show part<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">PartitionName=highmem<br>
AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL<br>
AllocNodes=ALL Default=NO QoS=N/A<br>
DefaultTime=02:00:00 DisableRootJobs=NO ExclusiveUser=NO GraceTime=0 Hidden=NO<br>
MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED<br>
Nodes=intel-00[01-20]<br>
PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=EXCLUSIVE<br>
OverTimeLimit=NONE PreemptMode=REQUEUE<br>
State=UP TotalCPUs=320 TotalNodes=20 SelectTypeParameters=NONE<br>
JobDefaults=(null)<br>
DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>