We are trying to design the charging and accounting system for our new institutional HPC facility and I'm having difficulty understanding exactly how we can use sacctmgr to achieve what we need.
Until now, our previous HPC facilities have all operated as free delivery and we have not needed to track costs by user/group/project. Account codes have been purely optional.
However, our new facility will be split into various resource types, with free partitions and paid/priority/reserved partitions across those resource types.
All jobs will need to be submitted with an account code.
For users submitting to 'free' partitions we don't need to track resource units against a balance, but the submitted account code would still be used for reporting purposes (i.e. "free resources accounted for % of all use by this project in August-September").
When submitting to a 'paid' partition, the account code needs to be checked to ensure it has a positive balance (or a balance that will not go past some negative threshold).
Each of the 'paid' partitions may (will) have a different resource unit cost. A simple example:
- Submit to a generic CPU paid partition -- 1 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a high-speed, non-blocking CPU paid partition -- 2 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a GPU paid partition -- 4 resource unit/token/credit/£/$ per allocated GPU card, per hour of compute
We need to have *one* pool of resource units/tokens/credits per account - let's say 1000 credits, and a group of users may well decide to spend all of their credits on the generic CPU partition, all on the GPU partition, or some mixture of the two.
So in the above examples, assuming one user (or group of users sharing the same account code) submit a 2 hour job to all three partitions, their one, single account code should be charged:
- 2 units for the generic CPU partition - 4 units for the job on the low latency partition - 8 units for the gpu partition. - A total of 14 credits removed from their single account code
Is this feasible to achieve without having to allocate credits to each of the partitions for an account, or creating a QOS variant for each and every combination of account and partition?
John Snowdon Senior Research Infrastructure Engineer (HPC)
Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://rse.ncldata.dev/
So, the overall answer is yes. But there's a lot to it. I can't detail everything we did to get where you're talking, but let me try to hit some high points. Basically, google anything I'm about to mention to get a fuller explanation:
* You can set the "cost" of things per partition via the TRESBillingWeights. That includes CPUs, mem, gpus. * You'll need to define anything other than cpu/mem/nodes that you'll want to bill for via AccountingStorageTRES. So, all your GPUs would be specified here. * Slurm provides a somewhat generic "billing" concept. So, for each minute of job time, you can indicate how much "billing" they've accumulated. * You can see how much billing they've accumulated over the life of the account via: sshare -a -h -p -o account,user,GrpTRESRaw,GrpTRESMins. The GrpTRESRaw contains a "billing=<number>". That's the number. There will be values for cpu=, mem=...and those are also indicators of how much they've used. But the billing is where you can set the rates via the TRESBillingWeights. * You can set how much billing they have available to use via: sacctmgr modify account set grptresmins=billing=<number> where account=<account>. * You can keep people from running jobs if they wouldn't have enough "billing" available by setting AccountingStorageEnforce=safe * NOTE: The GrpTRESRaw billing value just keeps going up. If the GrpTRESRaw value is higher than what is in GrpTresMins=billing...then they won't be able to run anything that could "cost" them some billing. In other words....you don't set their GrpTresMins=billing=1000 to give them 1000 "credits". Instead, you'd take the value from GrpTRESRaw billing, and ADD 1000 to that. Then set GrpTresMins=billing to that new value. THEN they have 1000 to spend.
I HIGHLY recommend setting up a test system and messing around with values to see what happens. That helps a lot better to understand how it works. I found the documentation not very helpful in UNDERSTANDING how it works. I had to trial/error it to get it.
Hope that helps. Bottom line...yes, it sounds like slurm can do most everything I saw you wanted to do.
Rob
________________________________ From: John Snowdon via slurm-users slurm-users@lists.schedmd.com Sent: Friday, October 18, 2024 4:56 AM To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Tracking costs - one single pool of credits, variable costs per partition
We are trying to design the charging and accounting system for our new institutional HPC facility and I'm having difficulty understanding exactly how we can use sacctmgr to achieve what we need.
Until now, our previous HPC facilities have all operated as free delivery and we have not needed to track costs by user/group/project. Account codes have been purely optional.
However, our new facility will be split into various resource types, with free partitions and paid/priority/reserved partitions across those resource types.
All jobs will need to be submitted with an account code.
For users submitting to 'free' partitions we don't need to track resource units against a balance, but the submitted account code would still be used for reporting purposes (i.e. "free resources accounted for % of all use by this project in August-September").
When submitting to a 'paid' partition, the account code needs to be checked to ensure it has a positive balance (or a balance that will not go past some negative threshold).
Each of the 'paid' partitions may (will) have a different resource unit cost. A simple example:
- Submit to a generic CPU paid partition -- 1 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a high-speed, non-blocking CPU paid partition -- 2 resource unit/token/credit/£/$ per allocated cpu, per hour of compute
- Submit to a GPU paid partition -- 4 resource unit/token/credit/£/$ per allocated GPU card, per hour of compute
We need to have *one* pool of resource units/tokens/credits per account - let's say 1000 credits, and a group of users may well decide to spend all of their credits on the generic CPU partition, all on the GPU partition, or some mixture of the two.
So in the above examples, assuming one user (or group of users sharing the same account code) submit a 2 hour job to all three partitions, their one, single account code should be charged:
- 2 units for the generic CPU partition - 4 units for the job on the low latency partition - 8 units for the gpu partition. - A total of 14 credits removed from their single account code
Is this feasible to achieve without having to allocate credits to each of the partitions for an account, or creating a QOS variant for each and every combination of account and partition?
John Snowdon Senior Research Infrastructure Engineer (HPC)
Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Frse.ncldat...https://rse.ncldata.dev/
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-
Groner, Rob -
John Snowdon