<html style="direction: ltr;">
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<style type="text/css">body p { margin-bottom: 0cm; margin-top: 0pt; } </style>
</head>
<body style="direction: ltr;" bidimailui-charset-is-forced="true"
bgcolor="#FFFFFF" text="#000000">
This solution is even better.<br>
I am actually using pestat for my (as admin) needs.<br>
But I originally asked the question in order to enhance the ability
of slurm_exporter which is a client side code for prometheus/grafana
that export slurm statistics to be read as graphs.<br>
for this need squeue -o %b is enough<br>
But I am sure there is a need for pestat to print the gres info as
well, you already atleast helping yair and myself.<br>
<br>
Thanks, Nadav<br>
<div class="moz-cite-prefix">On 13/02/2018 17:41, Ole Holm Nielsen
wrote:<br>
</div>
<blockquote
cite="mid:0a8f137c-e17e-f078-31bc-877fe22f7f95@fysik.dtu.dk"
type="cite">On 02/13/2018 08:13 AM, Nadav Toledo wrote:> Does
anyone know of way to get amount of idle gpu per node or for all
<br>
<blockquote type="cite">cluster ?
<br>
<br>
sinfo -o %G gives the total amount of gres resource for each
node. Is there a way to get the idle amount same as you can get
for cpu (%C)?
<br>
Perhaps if one use lock file like /dev/nvidia# for each gpu you
can check their states?
<br>
</blockquote>
<br>
I think printing the GRES usage for nodes is a neat idea. So I've
added a flag "-G" to my pestat command so that the GRES usage for
each job on each node is printed. The squeue command can print
GRES usage using -o %b.
<br>
<br>
Could you give pestat a try to see if it fits your needs:
<br>
<a class="moz-txt-link-freetext" href="https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat">https://github.com/OleHolmNielsen/Slurm_tools/tree/master/pestat</a>
<br>
<br>
Just run "pestat -G" on your Slurm cluster.
<br>
<br>
At the moment pestat doesn't print a column of total configured
GRES in the node, but this could be added if there is interest.
<br>
<br>
Please send me feedback and comments about pestat.
<br>
<br>
/Ole
<br>
<br>
</blockquote>
<br>
</body>
</html>