[slurm-users] [External] Re: Partition question

[Ole Holm Nielsen]

Some examples are here:
https://wiki.fysik.dtu.dk/niflheim/Slurm_accounting#quality-of-service-qos

/Ole

On 19-12-2019 19:30, Prentice Bisbal wrote:
> 
> On 12/19/19 10:44 AM, Ransom, Geoffrey M. wrote:
>>
>> The simplest is probably to just have a separate partition that will 
>> only allow job times of 1 hour or less.
>>
>> This is how our Univa queues used to work, by overlapping the same 
>> hardware. Univa shows available “slots” to the users and we had a lot 
>> of confused users complaining about all those free slots (busy slots 
>> in the other queue) while their jobs sat on the queue and new users 
>> confused as to why their jobs were being killed after 4 hours. I was 
>> able to move the short/long behavior to job classes and use RQSes and 
>> have one queue.
>>
>> While slurm isn’t showing users unused resources I am concerned that 
>> going back to two queues (partitions) will cause user interaction and 
>> adoption problems.
>>
>>          It all depends on what best suits the specific needs.
>>
>> Is there a way to have one partition that holds aside a small 
>> percentage of resources for jobs with a runtime under 4 hours, i.e. 
>> jobs with long runtimes cannot tie up 100% of the resources at one 
>> time? Some kind of virtual partition that feeds into two other 
>> partitions based on runtime would also work. The goal is that users 
>> can continue to post jobs to one partition but the scheduler won’t let 
>> 100% of the compute resources get tied up with mutli-week long jobs.
>>
> The way to do this is with Quality of Service (QOS) in Slurm. When 
> creating a QOS, you can specify the max. number of tasks a QOS can use. 
> Create a QOS for the longer running jobs and set the MaxGrpTRES so that 
> the number of CPUs is less that 100% of your cluster. Create a QOS for 
> the shorter jobs with a shorter time limit (MaxWall).
> 
> Once the QOSes are setup, you can instruct your users to specify the 
> proper QOS when submitting a job, or edit the job_submit.lua script to 
> look at the time limit specified, and assign/override the QOS based on 
> that.