[slurm-users] Calculate the GPU usages

[Paul Raines]

Doesn't add up for me either

# sacct -X -P --format=jobid,partition,ElapsedRaw,state,AllocTRES 
--starttime=2021-08-01 --endtime=2021-08-31 --user=jl1103  | grep gpu=

470069|1|FAILED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1
470351|1|FAILED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1
470352|1|FAILED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1
470359|4|FAILED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1
470370|515|CANCELLED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1
470395|1140|FAILED|billing=9,cpu=2,gres/gpu=2,mem=40G,node=2
470731|57875|FAILED|billing=4,cpu=1,gres/gpu=1,mem=20G,node=1

That is (1+1+1+4+515+2*1140+57875)/60 = 1011 minutes

# sreport cluster AccountUtilizationByUser start=2021-08-01 
end=2021-08-31 user=jl1103 --tres="gres/gpu"
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2021-08-01T00:00:00 - 2021-08-30T23:59:59 (2592000 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster     Account     Login     Proper Name      TRES Name     Used
------- ----------- --------- --------------- -------------- --------
    mlsc     *******    jl1103     ***********       gres/gpu      390


In slurm.conf for the partition all these jobs ran on I have
TRESBillingWeights="CPU=1.24,Mem=0.02G,Gres/gpu=3.0"  if that effects
the sreport number somehow -- but then I would expect sreport's number
to simply be 3x the sacct number then


-- Paul Raines (http://help.nmr.mgh.harvard.edu)