[slurm-users] Updated "pestat" tool for printing Slurm nodes status including GRES/GPU

[Ryan Novosielski]

Hi Ole,

Thanks again for your great tools!

Is something expected to have broken this script for older versions of Slurm somehow? A version we have with a file time of 1/19/21 will show job IDs and users for a given node, but the version you released yesterday does not seem to (we may have missed versions in the middle, so it may not be this version that did it):

Older: 

[root at amarel1 pestat]# ./pestat -F -w slepner080
Print only nodes that are flagged by * (RED nodes)
Select only nodes in hostlist=slepner080
Hostname       Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                            State Use/Tot              (MB)     (MB)  JobId User ...
slepner080           main*      mix  22  24    1.07*   128000   116325  17036194 mt1044 17032319 as2654 17039145 vs670  

Current:

[root at amarel1 pestat]# ~novosirj/bin/pestat -F -w slepner080
Print only nodes that are flagged by * (RED nodes)
Select only nodes in hostlist=slepner080
Hostname         Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                              State Use/Tot  (15min)     (MB)     (MB)  JobID User ...
slepner080           main*     mix   22  24    1.07*   128000   116325   

You can see Joblist and JobID User are not present.

--
#BlackLivesMatter
____
|| \\UTGERS,  	 |---------------------------*O*---------------------------
||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
     `'

> On Dec 13, 2021, at 7:09 AM, Ole Holm Nielsen <Ole.H.Nielsen at fysik.dtu.dk> wrote:
> 
> Hi Slurm users,
> 
> I have updated the "pestat" tool for printing Slurm nodes status with 1 line per node including job info.  The download page is https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat
> (also listed in https://slurm.schedmd.com/download.html).
> 
> Improvements:
> 
> * The GRES/GPU output option "pestat -G" now prints the job gres/gpu information as obtained from squeue's tres-alloc output option, which should contain the most correct GRES/GPU information.
> 
> If you have a cluster with GPUs, could you try out the latest version and send me any feedback?
> 
> Thanks to René Sitt for helpful suggestions and testing.
> 
> The pestat tool can print a large variety of node and job information, and is generally useful for monitoring nodes and jobs on Slurm clusters.  For command options and examples please see the download page.  My own favorite usage is "pestat -F".
> 
> Thanks,
> Ole
> 
> -- 
> Ole Holm Nielsen
> PhD, Senior HPC Officer
> Department of Physics, Technical University of Denmark
>