[slurm-users] slurmdbd not showing job accounting

[Douglas Jacobsen]

Sreport shows data that is summarized hourly. Restarting slurmdbd can delay
this process.  If some jobs are missing end records it can massively slow
the process because it may need to pick a much earlier start time in the
past to summarize.

Sacctmgr show runawayjobs can help identify if you are in this situation

On Sun, Oct 14, 2018 at 2:05 PM Dave Botsch <botsch at cnf.cornell.edu> wrote:

> This seems to reflect what I am seeing. Someone earlier mentioned
> multiple restarts of slurmdbd... those restarts never made data appear
> unless right around on the hour.
>
> It's as if instead of data getting sent right through slurmdbd that
> something in slurmdbd is just doing an hourly check of the text based
> sacct records (which I don't understand why those are even there if not
> configured in slurm.conf).
>
> Thanks.
>
> On Sun, Oct 14, 2018 at 12:08:10PM +0100, Antony Cleave wrote:
> > I have noticed on several clusters that sreport can be upto one hour out
> of
> > date i.e. it will update on the hour every hour.
> >
> > sacct does not behave this way and is always up to date.
> >
> > I cannot see this stated in the docs or see any config settings to
> control
> > this but it happens on the last 17.02 cluster I checked.
> >
> > Antony
> >
> > On 14 Oct 2018 11:58, "Steven Dick" <kg4ydw at gmail.com> wrote:
> >
> > It is documented that you need to create the cluster in the database.
> >
> > It is not documented that the accounting system won't work until you
> > restart slurmdbd multiple times before it starts collecting accounting
> > records.
> >
> > Also, none of the necessary restarts are needed on an upgrade -- only
> > when slurm is initialized for a new cluster.
> >
> >
> > On Sun, Oct 14, 2018 at 4:12 AM Ole Holm Nielsen
> > <Ole.H.Nielsen at fysik.dtu.dk> wrote:
> > > Correct, and this is documented in the Slurm accounting setup page:
> > > https://slurm.schedmd.com/accounting.html#database-configuration
> > >
>
> --
> ********************************
> David William Botsch
> Programmer/Analyst
> @CNFComputing
> botsch at cnf.cornell.edu
> ********************************
>
> --
Sent from Gmail Mobile
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20181014/a7e00e6a/attachment.html>