[slurm-users] How to check the percent cpu of a job?

[Ryan Novosielski]

Olm’s “pestat” script does allow you to get similar information, but I’m interested to see if indeed there’s a better answer. I’ve used his script for more or less the same reason, to see if the jobs are using the resources they’re allocated. They show at a node level though, and then you have to look closer. For example:

Print only nodes that are flagged by * (RED nodes)
Hostname       Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                            State Use/Tot              (MB)     (MB)  JobId User ...

  gpu003            oarc     drng*  8  12   58.06*    64000    24507  82565618 yc567  
...
 hal0027          kopp_1    alloc  28  28    8.64*   128000   115610  82591085 mes373 82595703 aek119 

You can see, both of the above are examples of jobs that have allocated CPU numbers that are very different from the ultimate CPU load (the first one using way more than allocated, though they’re in a cgroup so theoretically isolated from the other users on the machine), and the second one asking for all 28 CPUs but only “using” ~8 of them.

If you’re using cgroups, it would seem to me that there must also be a way to see the output of “top” for just a group, or at least something similar. systemd-cgtop does more or less that, but doesn’t seem to show exactly what you’d want here:

Path                                                                                                                     Tasks   %CPU   Memory  Input/s Output/s

/                                                                                                                          306  900.6     9.8G        -        -
/slurm                                                                                                                       -      -     3.7G        -        -
/slurm/uid_140780                                                                                                            -      -     3.0G        -        -
/slurm/uid_140780/job_82591085                                                                                               -      -     3.0G        -        -
/slurm/uid_142473                                                                                                            -      -   374.7M        -        -
/slurm/uid_142473/job_82595703                                                                                               -      -   374.7M        -        -

…CPU only being shown as an aggregate at the top level (sorry about the formatting).

> On Nov 21, 2018, at 1:27 PM, 宋亚磊 <songyl11 at lzu.edu.cn> wrote:
> 
> Hi Jing, thank you! 
> 
> The following command show us the cpu load of the node,
> 
> $ scontrol show node  <nodename_of_job> | grep CPULoad
> 
> but I want the percent cpu of the job, like top or ps.
> For examplt, a job allocated 10 cpus, but it just use 2, so the percent
> cpu should be 200%, not be 1000%, I want konw this.
> 
> Anyway, thank you again, Jing.
> 
> Best regards,
> Yalei
> 
>> -----原始邮件-----
>> 发件人: "Jing Gong" <gongjing at kth.se>
>> 发送时间: 2018-11-22 02:04:59 (星期四)
>> 收件人: "Slurm User Community List" <slurm-users at lists.schedmd.com>
>> 抄送: 
>> 主题: Re: [slurm-users] How to check the percent cpu of a job?
>> 
>> Hi,
>> 
>>> How to check the percent cpu of a job in slurm? 
>> 
>> We use command "scontrol" likes
>> 
>> $ scontrol show node  <nodename_of_job> | grep CPULoad
>> ...
>>   CPUAlloc=48 CPUErr=0 CPUTot=48 CPULoad=25.32
>> ...
>> 
>> Regards, Jing 
>> 
>> ________________________________________
>> From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of 宋亚磊 <songyl11 at lzu.edu.cn>
>> Sent: Wednesday, November 21, 2018 18:51
>> To: slurm-users at lists.schedmd.com
>> Subject: [slurm-users] How to check the percent cpu of a job?
>> 
>> Hello everyone,
>> 
>> How to check the percent cpu of a job in slurm? I tried sacct, sstat, squeue, but I can't find that how to check.
>> Can someone help me?
>> 
>> Best regards,
>> Yalei
>> 

--
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'