[slurm-users] Calculate the GPU usages

[Jeherul Islam]

Hi Jürgen,

No reservation was done during that time.

On Wed, Sep 1, 2021 at 7:11 PM Juergen Salk <juergen.salk at uni-ulm.de> wrote:

> Dear Jeherul,
>
> sacct is for job accounting, sreport for cluster usage accounting.
>
> Did you maybe had any resource reservations for this user in place during
> that period of time?
>
> To my very best knowledge, resource reservations for one or more users
> do count in terms of cluster usage as reported by sreport for that
> user(s),
> even if there were no jobs submitted into the reservation, while sacct
> reports the resources actually allocated by jobs but does not count idle
> reservation time.
>
> Best regards,
> Jürgen
>
>
> * Jeherul Islam <jeherul at gmail.com> [210901 18:24]:
> > Dear Loris,
> >
> > When we grep it by the user name "j.mira" it will strike out the multiple
> > counts. Again sacct is showing fewer gpu minutes than sreport.
> >
> > On Wed, 1 Sep, 2021, 6:03 PM Loris Bennett, <loris.bennett at fu-berlin.de>
> > wrote:
> >
> > > Dear Jeherul,
> > >
> > > Jeherul Islam <jeherul at gmail.com> writes:
> > >
> > > > Dear Loris,
> > > >
> > > > Thanks for your reply. Here is the output for the same period but the
> > > result is not matching.
> > > >
> > > > #sacct --account=chemistry
> > > --format=jobid,user,ElapsedRaw,state,AllocGRES,ncpus
> --starttime=2021-05-01
> > > --endtime=2021-08-31  | grep j.mira| grep gpu| awk '{sum += $3} END
> {print
> > > sum}'
> > >
> > > I think you need the option '-X' for 'sacct'.  This will give you one
> > > line per job rather than including the steps.  Without '-X' you are
> > > counting the usage multiple times for each job.
> > >
> > > Cheers,
> > >
> > > Loris
> > >
> > >
> > > > 6835053          (6835053/60 = 113917 )
> > > >
> > > > # sreport cluster AccountUtilizationByUser cluster=**** user=j.mira
> > > start=2021-05-01 end=2021-08-31 --tres="gres/gpu"
> > > >
> > >
> --------------------------------------------------------------------------------
> > > > Cluster/Account/User Utilization 2021-05-01T00:00:00 -
> > > 2021-08-30T23:59:59 (10540800 secs)
> > > > Usage reported in TRES Minutes
> > > >
> > >
> --------------------------------------------------------------------------------
> > > >   Cluster         Account     Login     Proper Name      TRES Name
> > >  Used
> > > > --------- --------------- --------- --------------- --------------
> > > --------
> > > > ********       chemistry    j.mira          j.mira       gres/gpu
> > >  149434
> > > >
> > > > On Wed, Sep 1, 2021 at 5:27 PM Loris Bennett <
> loris.bennett at fu-berlin.de>
> > > wrote:
> > > >
> > > >  Dear Jeherul,
> > > >
> > > >  Jeherul Islam <jeherul at gmail.com> writes:
> > > >
> > > >  > Dear All,
> > > >  >
> > > >  > Please share the correct way of calculating the GPU usages.
> > > >  > I am confused with sreport and sacct cmd. I am getting a different
> > > result.
> > > >  >
> > > >  > # sreport cluster AccountUtilizationByUser cluster=****
> user=j.mira
> > > start=2021-05-01 end=2021-08-31 --tres="gres/gpu"
> > > >
> > > >  Here you have:
> > > >
> > > >    end=2021-08-31
> > > >
> > > >  >
> > >
> --------------------------------------------------------------------------------
> > > >  > Cluster/Account/User Utilization 2021-05-01T00:00:00 -
> > > 2021-08-30T23:59:59 (10540800 secs)
> > > >  > Usage reported in TRES Minutes
> > > >  >
> > >
> --------------------------------------------------------------------------------
> > > >  >   Cluster         Account     Login     Proper Name      TRES Name
> > >  Used
> > > >  > --------- --------------- --------- --------------- --------------
> > > --------
> > > >  > ****       chemistry    j.mira          j.mira       gres/gpu
> > >  149434
> > > >  >
> > > >  > # sacct --account=chemistry
> > > --format=jobid,user,ElapsedRaw,state,AllocGRES,ncpus
> --starttime=2021-05-01
> > > --endtime=2021-08-01  | grep j.mira| grep gpu| awk '{sum += $3} END
> {print
> > > sum}'
> > > >
> > > >  whereas here you have
> > > >
> > > >    --endtime=2021-08-01
> > > >
> > > >  > 4957060
> > > >  >
> > > >  > Please share the correct way.
> > > >  >
> > > >  > With Thanks and regards
> > > >
> > > >  so, without having checked your sacct/awk logic I would not expect
> the
> > > results to be the same.
> > > >
> > > >  Cheers,
> > > >
> > > >  Loris
> > > >
> > > >  --
> > > >  Dr. Loris Bennett (Hr./Mr.)
> > > >  ZEDAT, Freie Universität Berlin         Email
> > > loris.bennett at fu-berlin.de
> > > --
> > > Dr. Loris Bennett (Hr./Mr.)
> > > ZEDAT, Freie Universität Berlin         Email
> loris.bennett at fu-berlin.de
> > >
> > >
>
> --
> GPG A997BA7A | 87FC DA31 5F00 C885 0DC3  E28F BD0D 4B33 A997 BA7A
>
>

-- 
Jeherul Islam
Technical Officer Grade I
Data Centre and High Performance Computing
Computer Centre
Indian Institute of Technology Guwahati
Guwahati-39
India
Office No :+91-361-258-3353
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210901/750b48bc/attachment-0001.htm>