[slurm-users] Getting usage reporting from sacct/sreport

[Juergen Salk]

Hi Thomas,

I think sreport should actually do what you want out of the box if you
have permissions to retrieve that information for other users than
yourself.  

In my understanding, sacct is meant for individual job and job step
accounting while sreport is more suitable for aggregated cluster usage
accounting. Thus, sreport also accounts for reservation hours which
sacct does not.

sreport should also be able to report on consumed GRES-hours, such as
GPU hours in your case, but you'll probably have to use '-T' option in
order to include that information to the report. 

In case it matters, our AccountingStorageTRES looks like that:

AccountingStorageTRES=gres/scratch,gres/gpu 

(We also account for local scratch space allocations as a GRES.) 

These are the commands that we usually point our users to when 
they ask for their historical ressource utilization:

  https://wiki.bwhpc.de/e/BwForCluster_JUSTUS_2_Slurm_HOWTO#How_to_retrieve_historical_resource_usage_for_a_specific_user_or_account.3F

(But omit 'user=<username>' or 'account=<account>' for a report on all 
users or accounts.) 

Hope that helps. 

Best regards
Jürgen


* Thomas Arildsen <thomas at arildsen.org> [230325 16:18]:
> I am experimenting with getting information from a Slurm cluster on how
> many resources each user has been consuming. I would like to get
> accumulated amount of CPU and GPU time over specified periods. GPU
> resources reported by type of GPU would be extra helpful.
> I am currently looking at sacct where I try options like:
> 
> sacct -a --starttime=2023-03-21T00:00 --
> format="user,totalcpu,tresusageintot%100"
> 
> "tresusageintot" shows me:
> "cpu=00:00:20,energy=0,fs/disk=0,mem=0,pages=3465,vmem=285140K ", so
> GPU information does not seem to be included and I have found no other
> option that can tell me.
> Also, it shows me individual job steps which I would really just like
> to aggregate. In fact I would just like to aggregate per user and
> ignore individual jobs.
> 
> I have also tried `sreport`, but I cannot really get anything useful
> out of it at the user level. For example:
> 
> sreport user TopUsage
> -----------------------------------------------------------------------
> ---------
> Top 10 Users 2023-03-21T00:00:00 - 2023-03-21T23:59:59 (86400 secs)
> Usage reported in CPU Minutes
> -----------------------------------------------------------------------
> ---------
>  Cluster     Login     Proper Name         Account      Used   Energy 
> --------- --------- --------------- --------------- --------- --------
> 
> It just gives me an empty table with no user information. I am guessing
> something is not configured right here to be storing that data.
> 
> I have "AccountingStorageTRES=gres/gpu" in slurm.conf. I am not sure
> what more I should perhaps put here.
> 
> I hope someone can advise on what I am missing here and how I can best
> get the usage stats I am hoping for.
> Best regards,
> 
> Thomas
> 
> -- 
> Special Consultant | CLAAUDIA
> 
> Phone: (+45) 9940 9844 | Email: tari at its.aau.dk | Web:
> https://www.claaudia.aau.dk/
> Aalborg University | Fredrik Bajers Vej 1, A1.65, 9220 Aalborg Ø,
> Denmark

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4965 bytes
Desc: not available
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230326/adc86fac/attachment.bin>