<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hey Andrew,<div class=""><br class=""></div><div class="">I don’t have any specific examples I can share right this second, I’ll look into making it shareable, but my solution was to throw some basic bash scripts into cron to scrap and ship into influx.</div><div class=""><br class=""></div><div class="">I have one script that looks at sinfo, parsing out AIOT state for nodes and CPUs, and then a very ugly, hacky sed/cut/awk to scrape GPU usage; as well as squeue to see jobs per state; both of these per partition and cluster.</div><div class="">I have another script that is basic sreport parsing for the tres/gres I care about, so that I can get a somewhat birdseye trend of utilization over time.</div><div class=""><br class=""></div><div class="">There’s likely to be something far, far better for this, but it was a quick and dirty solution to get something visible with existing tooling (Grafana/influx).</div><div class=""><br class=""></div><div class="">Reed<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 11, 2023, at 6:43 PM, Andrew Elwell <<a href="mailto:andrew.elwell@gmail.com" class="">andrew.elwell@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class=""><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Fri, 2 June 2023, 22:03 Jörg Striewski, <<a href="mailto:striewski@ismll.de" class="">striewski@ismll.de</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Hi, we use grafana with influx, it is easy to install and works fine</div></blockquote></div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Hi Jörg,</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Are your slurm to influx scripts publicly available anywhere? I do something similar for squeue via python subprocess to call</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">squeue -M all -a -o "%P,%a,%u,%D,%q,%T,%r"</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">And some sinfo calls for node/cpu usage:</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">sinfo -M {} -o "%P,%a,%F"</div><div dir="auto" class="">sinfo -M {} -o "%%R,%a,%C,%B,%z"</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">But I'd be interested to see what other places do. Perhaps some examples could be gathered for Ole's wiki?</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Andrew</div><div dir="auto" class=""><br class=""></div><div class="gmail_quote" dir="auto"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
</blockquote></div></div>
</div></blockquote></div><br class=""></div></body></html>