[slurm-users] job priority keeping resources from being used?

[c b]

I tried setting a 5 minute time limit on some low resource jobs, and one
hour on high resource jobs, but my 5 minute jobs are still waiting behind
the hourlong jobs.

Can you suggest some combination of time limits that would work here?



On Fri, Nov 1, 2019 at 11:08 AM c b <breedthoughts.www at gmail.com> wrote:

> On my low resource jobs I'm setting the time to 1 hour, and on my large
> ones I'm setting time=unlimited.
>
> Is the unlimited part the problem?  I have that setting because in my
> cluster there are some machines that come in and out during the day via
> reservations, and I want to keep these larger jobs from running on those
> machines.
>
>
>
>
>
> On Fri, Nov 1, 2019 at 10:56 AM Burian, John <
> John.Burian at nationwidechildrens.org> wrote:
>
>> Are you setting realistic job run times (sbatch –t <time>)?
>>
>>
>>
>> Slurm won’t backfill low priority jobs (with low resource requirements)
>> in front of a high priority job (blocked waiting on high resource
>> requirements) if it thinks the low priority jobs will delay the eventual
>> start of the high priority job. If all jobs are submitted with the same job
>> run time, then Slurm will never backfill, because as far as Slurm knows,
>> the low priority jobs will take longer to finish than just waiting for the
>> current running jobs to finish.
>>
>>
>>
>> John
>>
>>
>>
>>
>>
>> *From: *slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of
>> c b <breedthoughts.www at gmail.com>
>> *Reply-To: *Slurm User Community List <slurm-users at lists.schedmd.com>
>> *Date: *Friday, November 1, 2019 at 10:30 AM
>> *To: *"slurm-users at lists.schedmd.com" <slurm-users at lists.schedmd.com>
>> *Subject: *[slurm-users] job priority keeping resources from being used?
>>
>>
>>
>> [WARNING: External Email - Use Caution]
>>
>>
>>
>> Hi,
>>
>>
>>
>> Apologies for the weird subject line...I don't know how else to describe
>> what I'm seeing.
>>
>>
>>
>> Suppose my cluster has machines with 8 cores each.  I have many large
>> high priority jobs that each require 6 cores, so each machine in my cluster
>> runs one of each of these jobs at a time.  However, I also have lots of
>> small jobs that each require one core, and these jobs have low priority so
>> in my queue they are behind all my large jobs.
>>
>>
>>
>> In theory, these small jobs could slip in and run alongside the large
>> jobs, but I'm not seeing that happen.  So my machines have two cores
>> sitting idle when they could be doing work.  How do I configure slurm to
>> run these jobs better?
>>
>>
>>
>> thanks for any help.
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20191101/52173e37/attachment.htm>