[slurm-users] Job cancelled into the future

[Sarlo, Jeffrey S]

Do they show up as run away jobs?

sacctmgr show runawayjobs

If they do, it should give you the option to fix them.

Jeff

From: slurm-users <slurm-users-bounces at lists.schedmd.com> On Behalf Of Reed Dier
Sent: Tuesday, December 20, 2022 9:54 AM
To: Slurm User Community List <slurm-users at lists.schedmd.com>
Subject: [slurm-users] Job cancelled into the future

Hoping this is a fairly simple one.

This is a small internal cluster that we’ve been using for about 6 months now, and we’ve had some infrastructure instability in that time, which I think may be the root culprit behind this weirdness, but hopefully someone can point me in the direction to solve the issue.

I do a daily email of sreport to show how busy the cluster was, and who were the top users.
Weirdly, I have a user that seems to be able to use the same exact usage day after day after day, down to hundredth of a percent, conspicuously even when they were on vacation and claimed that they didn’t have job submissions in cron/etc.

So then, taking a spin of the scom tui <https://urldefense.com/v3/__https:/lists.schedmd.com/pipermail/slurm-users/2022-December/009514.html__;!!LkSTlj0I!HU994dhG-5tUYSafOMaTezByLfQVqq-CYpi_3KIx5Axh6N-cUxMFeVZDtk6fF54A-r_4HuXpc3RpvLtyeLibxTZO$> posted this morning, I then filtered that user, and noticed that even though I was only looking 2 days back at job history, I was seeing a job from August.

Conspicuously, the job state is cancelled, but the job end time is 1y from the start time, meaning its job end time is in 2023.
So something with the dbd is confused about this/these jobs that are lingering and reporting cancelled but still “on the books” somehow until next August.

╭──────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                          │
│  Job ID               : 290742                                                           │
│  Job Name             : $jobname                                                         │
│  User                 : $user                                                            │
│  Group                : $user                                                            │
│  Job Account          : $account                                                         │
│  Job Submission       : 2022-08-08 08:44:52 -0400 EDT                                    │
│  Job Start            : 2022-08-08 08:46:53 -0400 EDT                                    │
│  Job End              : 2023-08-08 08:47:01 -0400 EDT                                    │
│  Job Wait time        : 2m1s                                                             │
│  Job Run time         : 8760h0m8s                                                        │
│  Partition            : $part                                                            │
│  Priority             : 127282                                                           │
│  QoS                  : $qos                                                             │
│                                                                                          │
│                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────╯
Steps count: 0

Filter: $user         Items: 13

 Job ID      Job Name                             Part.  QoS         Account     User             Nodes                 State
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 290714      $jobname                             $part  $qos        $acct       $user            node32                CANCELLED
 290716      $jobname                             $part  $qos        $acct       $user            node24                CANCELLED
 290736      $jobname                             $part  $qos        $acct       $user            node00                CANCELLED
 290742      $jobname                             $part  $qos        $acct       $user            node01                CANCELLED
 290770      $jobname                             $part  $qos        $acct       $user            node02                CANCELLED
 290777      $jobname                             $part  $qos        $acct       $user            node03                CANCELLED
 290793      $jobname                             $part  $qos        $acct       $user            node04                CANCELLED
 290797      $jobname                             $part  $qos        $acct       $user            node05                CANCELLED
 290799      $jobname                             $part  $qos        $acct       $user            node06                CANCELLED
 290801      $jobname                             $part  $qos        $acct       $user            node07                CANCELLED
 290814      $jobname                             $part  $qos        $acct       $user            node08                CANCELLED
 290817      $jobname                             $part  $qos        $acct       $user            node09                CANCELLED
 290819      $jobname                             $part  $qos        $acct       $user            node10                CANCELLED

I’d love to figure out the proper way to either purge these jid’s from the accounting database cleanly, or change the job end/run time to a sane/correct value.
Slurm is v21.08.8-2, and ntp is a stratum 1 server, so time is in sync everywhere, not that multiple servers would drift 1 year off like this.

Thanks for any help,
Reed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20221220/b521ab8b/attachment-0001.htm>