[slurm-users] Fairshare +FairTree Algorithm + TRESBillingWeights

[Luke Yeager]

  *   Rawshare is only a representation of weight where higher value equal to higher priority ?
  *   The total of rawshare need not to be at 100 since it is not percentage?

Look at the output of this command on your cluster and things will probably become more clear:
sshare -a -format=cluster,account,user,partition,rawshare,normshare,normusage,levelfs


  *   Job from User7 will always run before User2 (including those in queue)

No, not "always." It depends on how much each user (actually, user-to-project association) has been utilizing the cluster recently. See "normusage" in the command above. If levelfs is >1, then the priority of the job will be boosted (because they have been "under-served" recently - that's how the manpage puts it).


  *   Is there a command to print out the billing weight is indeed as requested instead of the default cpu ?

Take a look at the output of this command. In particular, check out the value for GrpTRESMins for your "Association Records":
scontrol show assoc_mgr

Hope that helps!
Luke

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Yap, Mike
Sent: Wednesday, March 31, 2021 4:50 PM
To: slurm-users at schedmd.com
Subject: [slurm-users] Fairshare +FairTree Algorithm + TRESBillingWeights

External email: Use caution opening links or attachments

Hi All

Need some clarification on Fairshare (multifactor priority plugin) and FairTree Algorithm

If I read correctly, the current default for slurm is FairTree algorithm in which

  1.  Priority can set on various level
  2.  No fairshare-actual usage is being consider
  3.  Job submitted will run according to Priority instead of fairshare-actual usage
Questions -

  1.  Rawshare is only a representation of weight where higher value equal to higher priority ?
  2.  The total of rawshare need not to be at 100 since it is not percentage?
  3.  Referring to image below, am I right to believe

     *   Job from User7 will always run before User2 (including those in queue)
     *   Job from User3 will always run before User2 (including those in queue)
     *   Job from User7 will always run before User6 (including those in queue)
     *   Job from User3 will always run before User8 (including those in queue)
     *   Job from User2 will always run before User1 (including those in queue)
[cid:image001.png at 01D729F1.BFC86F00]


Only by configuring multifactor priority plugin, fairshare-usage will be consider in which (referring to above image again) in which it will be common for User1(adhoc user) to have job run before User4(user with massive clocktime)
Questions

  1.  In scenario if both user1 and user2 having the same fairshare value (sshare showing 0.001), do User4 will still have priority over User1 ?

Additional question

  1.  how do we enforce only user define in sacctmgr have the right to submit job ? restarted the services and system but user not in sacctmgr is still able to submit job
  2.  User with two account, how do they define which account to use for submission ?

TRESBillingWeights

If I wish to enable TRESBillingWeight (to use CPU:MEMORY:GPU) - referring to https://slurm.schedmd.com/tres.html<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslurm.schedmd.com%2Ftres.html&data=04%7C01%7Clyeager%40nvidia.com%7Cdcbc5e7a2ea94213b12a08d8f4a00971%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C1%7C637528315492276561%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=UQHn2dkXCw5UPFNWgp3yHu2dAH%2BEfhFm6cJ%2BE2QlYAY%3D&reserved=0>


  1.  Do I just include the following line to enable the option

     *   AccountingStorageTRES=gres/gpu
     *   TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=2.0"

  1.  Is there a command to print out the billing weight is indeed as requested instead of the default cpu ?

Many thanks
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210405/d41ae217/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 9105 bytes
Desc: image001.png
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210405/d41ae217/attachment-0001.png>