[slurm-users] Automatically converting jobs to regular priority when high-priority is exhausted

[Groner, Rob]

I didn't see this thread before, so maybe this has already been suggested...

When submitting jobs with sbatch, you could specify a list of partitions to use, and slurm will send the jobs to the partition with the earliest start/highest priority first, and if that gets "full" then it will send the remaining jobs to the next partition in the list.  You are talking about accounts, not partitions, but you also said you wanted to use just 1 account.  If you used 1 account and had the partitions determine priority (instead of the account), then that could work.

To be honest, I've never tried to do this...but I do know that's how it's SUPPOSED to work.

Rob

________________________________
From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Ebbert, Mark T. <Mark.Ebbert at uky.edu>
Sent: Thursday, August 3, 2023 5:37 AM
To: slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
Subject: Re: [slurm-users] Automatically converting jobs to regular priority when high-priority is exhausted

You don't often get email from mark.ebbert at uky.edu. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>

Hello,



Just wanted to give another go at this question.



If no one has input, is there somewhere I can submit feature requests? I didn’t see a way to do that on the schedmd website.



____________________________________________



Mark T. W. Ebbert, Ph.D.

Assistant Professor

University of Kentucky

Sanders-Brown Center on Aging

545 Lee T. Todd Jr. Building

Lexington, KY 40536

859.218.0125<tel:859-218-0125>

ebbertlab.com





From: Ebbert, Mark T. <Mark.Ebbert at uky.edu>
Date: Wednesday, July 26, 2023 at 6:49 PM
To: slurm-users at lists.schedmd.com <slurm-users at lists.schedmd.com>
Subject: Automatically converting jobs to regular priority when high-priority is exhausted

Hello,



I’m a SLURM user (not administrator) so I may not use the correct terms, but hopefully I can get my question across.



Ultimate question: Is there a way to have SLURM automatically convert job submissions from a “high-priority” account to regular priority when that account’s submissions have been maxed out? If not, is there a way to request such a feature?



More background: Essentially, my lab as two accounts that we can use on our campus cluster. One is given higher priority in the queue because we directly funded a certain number of compute nodes, while the other has regular priority. Once we exhaust the number of job units allotted to our high-priority account, any new jobs submitted from that account get straight-up rejected from the scheduler. We don’t want to have to track how many units we’ve used from a certain account and then switch to a different account. We’d like to always be able to submit under a single account and let SLURM first use our allotted high-priority units, but once those are used up, simply set the rest of the jobs as regular priority (and not reject them).



We use Nextflow to manage our job submissions, so in the middle of a large and complicated run, all of our jobs start getting rejected. Going through our scripts, etc. and changing accounts is a waste of time and error prone. It’s gotten to the point where we’re only submitting jobs under our regular account so we don’t have to deal with our jobs getting rejected, which defeats the purpose of buying into the high-priority to begin with.



Thanks for your help.



____________________________________________



Mark T. W. Ebbert, Ph.D.

Assistant Professor

University of Kentucky

Sanders-Brown Center on Aging

545 Lee T. Todd Jr. Building

Lexington, KY 40536

859.218.0125<tel:859-218-0125>

ebbertlab.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230803/ffa67ce4/attachment.htm>