[slurm-users] Getting usage reporting from sacct/sreport

[Mike Mikailov]

Hi Thomas et al,

I have just written a Linux shell script which does exactly what you are asking for.

Please use “—allocations” option in sacct command to generate aggregated resources usage per user.

You may also use awk Linux command to summarize all CPU usages. 

More advanced awk command may also summarize all GPU usages.

I have also placed the script on the GitHub but it is private now until we clear it for public.

Traceable resources normalization along with traceable resources weights are needed for more fair usage reports. in this case “billing” value represents combined (max or sum of individual traceable resources) billing unit. Note by default this values equals to the number of CPUs used. 

Thanks,
-Mike
USA

Sent from my iPhone

> On Mar 25, 2023, at 11:21 AM, Thomas Arildsen <thomas at arildsen.org> wrote:
> 
> I am experimenting with getting information from a Slurm cluster on how
> many resources each user has been consuming. I would like to get
> accumulated amount of CPU and GPU time over specified periods. GPU
> resources reported by type of GPU would be extra helpful.
> I am currently looking at sacct where I try options like:
> 
> sacct -a --starttime=2023-03-21T00:00 --
> format="user,totalcpu,tresusageintot%100"
> 
> "tresusageintot" shows me:
> "cpu=00:00:20,energy=0,fs/disk=0,mem=0,pages=3465,vmem=285140K ", so
> GPU information does not seem to be included and I have found no other
> option that can tell me.
> Also, it shows me individual job steps which I would really just like
> to aggregate. In fact I would just like to aggregate per user and
> ignore individual jobs.
> 
> I have also tried `sreport`, but I cannot really get anything useful
> out of it at the user level. For example:
> 
> sreport user TopUsage
> -----------------------------------------------------------------------
> ---------
> Top 10 Users 2023-03-21T00:00:00 - 2023-03-21T23:59:59 (86400 secs)
> Usage reported in CPU Minutes
> -----------------------------------------------------------------------
> ---------
> Cluster     Login     Proper Name         Account      Used   Energy 
> --------- --------- --------------- --------------- --------- --------
> 
> It just gives me an empty table with no user information. I am guessing
> something is not configured right here to be storing that data.
> 
> I have "AccountingStorageTRES=gres/gpu" in slurm.conf. I am not sure
> what more I should perhaps put here.
> 
> I hope someone can advise on what I am missing here and how I can best
> get the usage stats I am hoping for.
> Best regards,
> 
> Thomas
> 
> -- 
> Special Consultant | CLAAUDIA
> 
> Phone: (+45) 9940 9844 | Email: tari at its.aau.dk | Web:
> https://www.claaudia.aau.dk/
> Aalborg University | Fredrik Bajers Vej 1, A1.65, 9220 Aalborg Ø,
> Denmark