[slurm-users] Information about finished jobs

[Paul Raines]

I have been writing my own 'jobinfo' tool for users to see info on
a job in any state that is useful and readable by them.  Still
new to slurm and trying to wrap my head around the database info
and the effects of arrays and such.

A completed job output looks like this:

# jobinfo 357300
--------------------------------------------------
           JobID : 356847_361          | 356847_361.batch
         JobName : batch_compile_loraks.sh
            User : gr879
         Account : syhdiff
       Partition : basic
         ReqTRES : billing=1,cpu=1,mem=40G,node=1
       AllocTRES : billing=1,cpu=1,mem=40G,node=1
        NodeList : r440-19
          Submit : 2021-06-13T22:07:07
           Start : 2021-06-14T01:47:55 | 2021-06-14T01:47:55
             End : 2021-06-14T05:22:00 | 2021-06-14T05:22:00
       Timelimit : 2-00:00:00
         Elapsed : 03:34:05            | 03:34:05
         CPUTime : 03:34:05            | 03:34:05
       SystemCPU : 05:57.056           | 05:57.056
         UserCPU : 03:27:07            | 03:27:07
        TotalCPU : 03:33:04            | 03:33:04
     MaxDiskRead :                     | 109.25M
    MaxDiskWrite :                     | 1.08M
          MaxRSS :                     | 32529204K
       MaxVMSize :                     | 61834112K
           State : COMPLETED           | COMPLETED
        ExitCode : 0:0                 | 0:0
         WorkDir : /autofs/homes/002/gr879/matlab/ex_vivo/batch_code

and a typical RUNNING job looks like

# jobinfo 357304
--------------------------------------------------
           JobID : 357199_21           | 357199_21.batch
         JobName : batch_compile_multi_shell.sh
            User : gr879
         Account : syhdiff
       Partition : basic
         ReqTRES : billing=1,cpu=1,mem=12G,node=1
       AllocTRES : billing=1,cpu=1,mem=12G,node=1
        NodeList : r440-17
          Submit : 2021-06-14T00:31:11
           Start : 2021-06-14T01:47:55 | 2021-06-14T01:47:55
             End : Unknown             | Unknown
       Timelimit : 1-00:00:00
         Elapsed : 12:04:35            | 12:04:35
         CPUTime : 12:04:35            | 12:04:35
       SystemCPU : 00:00:00            | 00:00:00
         UserCPU : 00:00:00            | 00:00:00
        TotalCPU : 00:00:00            | 12:01:46
     MaxDiskRead :                     | 101176763
    MaxDiskWrite :                     | 1259187
          MaxRSS :                     | 5455M
       MaxVMSize :                     | 10823600K
           State : RUNNING             | RUNNING
        ExitCode : 0:0                 | 0:0
         WorkDir : /autofs/homes/002/gr879/matlab/ex_vivo/batch_code

Where unfortunately I have to give zeros on certain info
I cannot get yet.  My current issue is with that TotalCPU row
on running jobs.  I actually get that from AveCPU from sstat and
in the case above it looks right.  But in others it is just way off

# jobinfo 357305
--------------------------------------------------
           JobID : 357305              | 357305.batch
         JobName : sjob_185
            User : mjk2
         Account : circgp
       Partition : basic
         ReqTRES : billing=27,cpu=20,mem=370G,node=1
       AllocTRES : billing=27,cpu=20,mem=370G,node=1
        NodeList : r440-05
          Submit : 2021-06-14T01:44:56
           Start : 2021-06-14T05:02:10 | 2021-06-14T05:02:10
             End : Unknown             | Unknown
       Timelimit : 7-00:00:00
         Elapsed : 08:50:17            | 08:50:17
         CPUTime : 7-08:45:40          | 7-08:45:40
       SystemCPU : 00:00:00            | 00:00:00
         UserCPU : 00:00:00            | 00:00:00
        TotalCPU : 00:00:00            | 11:33.000
     MaxDiskRead :                     | 79699046
    MaxDiskWrite :                     | 17983
          MaxRSS :                     | 81357340K
       MaxVMSize :                     | 104992372K
           State : RUNNING             | RUNNING
        ExitCode : 0:0                 | 0:0
         WorkDir : /autofs/homes/002/mjk2

In this job the user asked for 20 cores, but I can see his
job is only one one core on the actual node so this is a big waste.
But that core is constantly going 100% so I would expect AveCPU
to be close to the Elapsed time but is is way less (11 minutes
instead of nearly 9 hours)

# /usr/bin/sstat -p -a --job=357305 --format=JobID,AveCPU
JobID|AveCPU|
357305.extern|213503982334-14:25:51|
357305.batch|11:33.000|

Any idea why this is?  Also, what is that crazy number for
AveCPU on 357305.extern?

-- Paul Raines (http://help.nmr.mgh.harvard.edu)



On Mon, 14 Jun 2021 2:45am, Ole Holm Nielsen wrote:

> On 6/14/21 8:26 AM, Gestió Servidors wrote:
>>  How can I get all information about a finished job in the same way as
>>  “scontrol show jobid=” when job is pending or running?
>
> Some minutes after job completion, you can only get the information which is 
> stored in the Slurm database.
>
> My script "showjob" in 
> https://github.com/OleHolmNielsen/Slurm_tools/tree/master/jobs shows all 
> available information for jobs in the queue as well as in the database.
>
> /Ole
>
>
>
>