[slurm-users] job priority keeping resources from being used?

[Burian, John]

Are you setting realistic job run times (sbatch –t <time>)?

Slurm won’t backfill low priority jobs (with low resource requirements) in front of a high priority job (blocked waiting on high resource requirements) if it thinks the low priority jobs will delay the eventual start of the high priority job. If all jobs are submitted with the same job run time, then Slurm will never backfill, because as far as Slurm knows, the low priority jobs will take longer to finish than just waiting for the current running jobs to finish.

John


From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of c b <breedthoughts.www at gmail.com>
Reply-To: Slurm User Community List <slurm-users at lists.schedmd.com>
Date: Friday, November 1, 2019 at 10:30 AM
To: "slurm-users at lists.schedmd.com" <slurm-users at lists.schedmd.com>
Subject: [slurm-users] job priority keeping resources from being used?

[WARNING: External Email - Use Caution]


Hi,

Apologies for the weird subject line...I don't know how else to describe what I'm seeing.

Suppose my cluster has machines with 8 cores each.  I have many large high priority jobs that each require 6 cores, so each machine in my cluster runs one of each of these jobs at a time.  However, I also have lots of small jobs that each require one core, and these jobs have low priority so in my queue they are behind all my large jobs.

In theory, these small jobs could slip in and run alongside the large jobs, but I'm not seeing that happen.  So my machines have two cores sitting idle when they could be doing work.  How do I configure slurm to run these jobs better?

thanks for any help.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20191101/5853a079/attachment.htm>