Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account.
I have two sets of physical nodes: 1. node01 - nodeXX: General Purpose Nodes. Everybody can use them 2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them.
Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts.
I have three sets of accounts (each can have child accounts): 1. "General" accounts: These are allowed to use all physical nodes. 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.
My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.
In the future, there might be more project specific nodes for other projects.
I would be grateful for any alternative solution or hints, how this could be solved optimally.
Thanks a lot! Thomas
Hi,
My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.
That is what we do. We also set a higher priority on the restricted access partition so jobs in it get queued faster. I don't know if it is the best way to do this though.
Thanks
Sean
--- Sean McGrath
Systems Administrator Research IT IT Services Trinity College Dublin
00 353 - (0)1 896 3725 https://www.tcd.ie/itservices/ https://www.tchpc.tcd.ie/http://www.tchpc.tcd.ie/ ________________________________ From: thomas.hartmann--- via slurm-users slurm-users@lists.schedmd.com Sent: Wednesday 21 May 2025 09:25 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Restrict and prioritize usage of certain nodes according to accounts
[External Email] This email originated outside of Trinity College Dublin. Do not click links or open attachments unless you recognise the sender and know the content is safe.
Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account.
I have two sets of physical nodes: 1. node01 - nodeXX: General Purpose Nodes. Everybody can use them 2. project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them.
Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts.
I have three sets of accounts (each can have child accounts): 1. "General" accounts: These are allowed to use all physical nodes. 2. "ForProfit" accounts: These absolutely must not use the project_A_* nodes 3. "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.
My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.
In the future, there might be more project specific nodes for other projects.
I would be grateful for any alternative solution or hints, how this could be solved optimally.
Thanks a lot! Thomas
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
"thomas.hartmann--- via slurm-users" slurm-users@lists.schedmd.com writes:
I have three sets of accounts (each can have child accounts):
- "General" accounts: These are allowed to use all physical nodes.
- "ForProfit" accounts: These absolutely must not use the project_A_* nodes
- "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.
My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.
That will work, but will not ensure that Project_A jobs first land on the projectt_A_* nodes (unless you give these nodes a lower weight - but then the General jobs will also start there first).
You could add a third partition with the project_A_* nodes, and only give Project_A access to it. You can then give this partition a higher PriorityTier than the other partitions, meaning that jobs asking for that partitions will be scheduled first. Users of project_A should then submit jobs with -p project_A_part,General_part, or you could add some logic in your job_submit.lua that makes sure this happens.
Users are associated with one or more accounts.
If you can avoid this ( I know that's a big "if"), avoid it! Humans being humans (lazy) and users begin users (copying existing work as much as possible), if a user has multiple accounts, there is a damn good chance they'll submit jobs to the wrong accounts eventually, messing up priority/fairshare and accounting/billing, which will eventually add up to headaches for you.
I know it's not always possible to eliminate it, and it can be needed/desirable to have this capability. If you can minimize it as much as possible, though, it will minimize headaches for you.
I did work at one place where a Slurm user could only belong to one account at a time to prevent those sort of issues.
Prentice
On 5/21/25 4:25 AM, thomas.hartmann--- via slurm-users wrote:
Hi, I'm going to have the following situation on my hands and I would like to ask for some suggestions how to solve this in slurm. We're talking about a cluster that is not yet operational so there are no real legacy configs that one needs to take into account.
I have two sets of physical nodes:
- node01 - nodeXX: General Purpose Nodes. Everybody can use them
- project_A_01 - project_A_XX: These nodes come with some restrictions regarding who can and who must not run jobs on them.
Users are organized into accounts, pseudohierarchically representing workgroups / projects etc within the university. For instance, there is going to be a "physics" account with "project_blackhole", "workgroup_miller" as children. Users are associated with one or more accounts.
I have three sets of accounts (each can have child accounts):
- "General" accounts: These are allowed to use all physical nodes.
- "ForProfit" accounts: These absolutely must not use the project_A_* nodes
- "Project_A" accounts: Their jobs should run first and foremost on the project_A_* nodes but they are also allowed to run on the node* nodes.
My first idea would be to create two partitions, one with all nodes in there and a second one with only those nodes that the "ForProfit" are allowed to use. Using "AllowAccounts" or "DenyAccounts" would implement the restriction I need.
In the future, there might be more project specific nodes for other projects.
I would be grateful for any alternative solution or hints, how this could be solved optimally.
Thanks a lot! Thomas