[slurm-users] [External] Re: Per-user TRES summary?

[Pacey, Mike]

Hi Ole (and Jeffrey),

Thanks for the pointer - those are some very useful scripts. I couldn't get showslurmlimits or showslurmjobs to get quite what I was after (it wasn't showing me memory usage). However, it pointed me in the right direction - the scontrol command. I can run the following:

scontrol show assoc_mgr flags=qos

and part of the output reads:

    User Limits
      [myuid]
	MaxJobsPU=N(2) MaxJobsAccruePU=N(0) MaxSubmitJobsPU=N(2)
	MaxTRESPU=cpu=80(2),mem=327680(1000),energy=N(0),node=N(1),billing=N(2),fs/disk=N(0),vmem=N(0),pages=N(0)

Which is exactly what I'm looking for. The values outside the brackets are the qos limit, and the values within are the current usage.

Regards,
Mike

-----Original Message-----
From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Ole Holm Nielsen
Sent: 28 November 2022 18:58
To: slurm-users at lists.schedmd.com
Subject: [External] Re: [slurm-users] Per-user TRES summary?

This email originated outside the University. Check before clicking links or attachments.

Hi Mike,

Would the "showuserlimits" tool give you the desired information?  Check out https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOleHolmNielsen%2FSlurm_tools%2Ftree%2Fmaster%2Fshowuserlimits&data=05%7C01%7Cpacey%40live.lancs.ac.uk%7Cbea74c16c0b34468c68908dad174dacc%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638052597059366026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9pKyND95SW9Z1E%2BjGPsGKUKwTZIqj3juwWGQ4d5AWRw%3D&reserved=0

/Ole


On 28-11-2022 16:16, Pacey, Mike wrote:
> Does anyone have suggestions as to how to produce a summary of a 
> user's TRES resources for running jobs? I'd like to able to see how 
> each user is fairing against their qos resource limits. (I'm looking 
> for something functionally equivalent to Grid Engine's qquota 
> command). The info must be in the scheduler somewhere in order for it 
> to enforce qos TRES limits, but as a SLURM novice I've not found any way to do this.
>
> To summarise TRES qos limits I can do this:
>
> % sacctmgr list qos format=Name,MaxTRESPerUser%50
>
>        Name                                          MaxTRESPU
>
> ---------- --------------------------------------------------
>
>      normal                                    cpu=80,mem=320G
>
> But to work out what a user is currently using in currently running 
> jobs, the nearest I can work out is:
>
> % sacct -X -s R --units=G -o User,ReqTRES%50
>
>       User                                            ReqTRES
>
> --------- --------------------------------------------------
>
>      pacey                   billing=1,cpu=1,mem=0.49G,node=1
>
>      pacey                   billing=1,cpu=1,mem=0.49G,node=1
>
> With a little scripting I can sum those up, but there might be a 
> neater way to do this?