I've found that when using sacct to track resource usage over specific time periods, it's helpful to include the --truncate option. Without it, jobs that started before the specified start time will have their entire runtime counted, including time outside the specified range. The --truncate option ensures that only the time within the defined period is included. Maybe this can explain some of the discrepancy you experience.

From: Passant Hafez via slurm-users <slurm-users@lists.schedmd.com>
Sent: Wednesday, May 21, 2025 18:48
To: 'slurm-users@schedmd.com' <slurm-users@schedmd.com>
Subject: [slurm-users] Slurm Reporting Difference between sreport and sacct

Hi all,

I was wondering if someone can help explaining this discrepancy.

I have different values for project gpu consumption using sreport vs sacct (+ some calculations)

This is an example that shows this:

sreport -t hours -T gres/gpu cluster AccountUtilizationByuser start=2025-04-01 end=2025-04-05 | grep project1234

gives 178

while

sacct -n -X --allusers --accounts=project1234 --start=2025-04-01 --end=2025-04-05 -o elapsedraw,AllocTRES%80,user,partition

gives

213480 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus

249507 billing=128,cpu=128,gres/gpu=8,mem=1000G,node=2 gpuplus

13908 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus

9552 billing=64,cpu=64,gres/gpu=4,mem=500G,node=1 gpuplus

4 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu

11 billing=16,cpu=16,gres/gpu=1,mem=200G,node=1 gpu

...

I will not bore you with the full output and its calculation, but the first job alone consumed 213480 seconds/60/60 * 8 gpus that's 474.4 gpu hours which is way more than the 178 hrs reported by sreport

Any clue why these are inconsistent? or how sreport calculated the 178 value?

All the best,

Passant