[slurm-users] Is it possible to set a default QOS per partition?
stackkorora at disroot.org
Mon Mar 1 21:24:46 UTC 2021
We have different node classes that we've set up in different
partitions. For example, we have our standard compute nodes in compute;
our GPU's in a gpu partition; and jobs that need to run for months go
into a long partition with a different set of machines.
For each partition, we have QOS to prevent any single user from
dominating the resources (set at a max of 60% of resources; not my call
- it's politics - I'm not going down that rabbit hole...).
Thus, I've got something like this in my slurm.conf (abbreviating to
save space; sorry if I trim too much).
PartitionName=compute [snip] AllowQOS=compute Default=YES
PartitionName=gpu [snip] AllowQOS=gpu Default=NO
PartitionName=long [snip] AllowQOS=long Default=NO
Then I have my QOS configured. And in my `sacctmgr dump cluster | grep
DefaultQOS` I have "DefaultQOS=compute".
All of that works exactly as expected.
This makes it easy/nice for my users to just do something like:
$ sbatch -n1 -N1 -p compute script.sh
They don't have to specify the QOS for compute and they like this.
However, for the other partitions they have to do something like this:
$ sbatch -n1 -N1 -p long --qos=long script.sh
The users don't like this. (though with scripts, I don't see the big
deal in just adding a new line...but you know... users...)
The request from the users is to make a default QOS for each partition
thus not needing to specify the QOS for the other partitions.
Because the default is set in the cluster configuration, I'm not sure
how to do this. And I'm not seeing anything in the documentation for a
scenario like this.
Anyone know how I can set a default QOS per partition?
Chesterton's fence and all... Is there a better way to accomplish what
we are attempting to do? I don't want a single QOS to limit across all
partitions. I need a per partition limit that restricts users to 60% of
resources in that partition.
More information about the slurm-users