In theory, if jobs are pending with “Priority”, one or more other jobs will be pending with “Resources”.

So a few questions:

1. What are the “Resources” jobs waiting on, resource-wise?
2. When are they scheduled to start?
3. Can your array jobs backfill into the idle resources and finish before the “Resources” jobs are scheduled to start?

From: Long, Daniel S. via slurm-users <slurm-users@lists.schedmd.com>
Date: Tuesday, September 24, 2024 at 11:47 AM
To: slurm-users@schedmd.com <slurm-users@schedmd.com>
Subject: [slurm-users] Jobs pending with reason "priority" but nodes are idle

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

---

Hi,

On our cluster we have some jobs that are queued even though there are available nodes to run on. The listed reason is “priority” but that doesn’t really make sense to me. Slurm isn’t picking another job to run on those nodes; it’s just not running anything at all. We do have a quite heterogeneous cluster, but as far as I can tell the queued jobs aren’t requesting anything that would preclude them from running on the idle nodes. They are array jobs, if that makes a difference.

Thanks for any help you all can provide.