<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Would be interested if anyone knows of any such scripts that already exist. <br>
<br>
<div id="AppleMailSignature"><span style="background-color: rgba(255, 255, 255, 0);">--<br>
____<br>
|| \\UTGERS, |---------------------------*O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="1">novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'</span></div>
<div><br>
On Jan 19, 2018, at 03:44, John Hearns <<a href="mailto:hearnsj@googlemail.com">hearnsj@googlemail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>Not specifically Slurm, but it can be useful to have alerts on jobs which either will never start or which are 'stalled'.</div>
<div>You might want to have an alert on jobs which (say) request more slots or nodes than physicall exist, so the users job will never run.</div>
<div>Or you can look for 'stalled' jobs where the CPU time used never increases.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 19 January 2018 at 08:56, Marcin Stolarek <span dir="ltr">
<<a href="mailto:stolarek.marcin@gmail.com" target="_blank">stolarek.marcin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>We're using icinga2 storing accounting data in influxdb for grafana dashboards. In terms of monitoring I prefere end-user functionality, so apart from services we also have a plugin that submits a jobs to cluster (to idle nodes, with a few minutes of deadline)
the job simply creates files on shared filesystem effectively monitoring slurmctl, slurmd, sssd, filesystems etc.
<br>
<br>
</div>
cheers,<br>
</div>
Marcin <br>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-01-19 5:44 GMT+01:00 Ryan Novosielski <span dir="ltr">
<<a href="mailto:novosirj@rutgers.edu" target="_blank">novosirj@rutgers.edu</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<span>> On Jan 18, 2018, at 4:34 PM, Lachlan Musicman <<a href="mailto:datakid@gmail.com" target="_blank">datakid@gmail.com</a>> wrote:<br>
><br>
> On 19 January 2018 at 07:29, Ryan Novosielski <<a href="mailto:novosirj@rutgers.edu" target="_blank">novosirj@rutgers.edu</a>> wrote:<br>
> Hi all,<br>
><br>
> Looked back at the mailing list to see if there was a question about this already. There was some mention of /using/ Nagios, but no real mention of specifics. What do people monitor with Nagios? We monitor, so far, slurmctld, slurmdbd, and MySQL, but there
are probably some others. Might be helpful to run “scontrol ping” for example, or similar, on our login nodes.<br>
><br>
> Does anyone have any plugins they’ve written or ideas they can share? Nagios Exchange doesn’t have anything with SLURM anywhere in the name.<br>
><br>
> Thanks!<br>
><br>
><br>
> Off the top of my head the only other two that I would want explicitly would be:<br>
> - ntp/chrony and their respective ntpd. Nodes go offline when the timing slides too far, especially if you are using Munge.<br>
> - authentication system - in our case ipa/sssd. Without that, even the queued jobs will fail.<br>
><br>
> We use Zabbix in house. I was under the impression that people were moving toward icingia2 over Nagios.<br>
<br>
</span>I wouldn’t mind moving to Icinga2 over Nagios, but really, it’s more or less a nicer version of the same thing, so I’d have the same question with Icinga2.<br>
<br>
Thanks for the NTP/Chrony tip though — if I get only that from this thread, it will have been worth it. That’s caused us trouble more than once. We do already monitor our LDAP, but SSSD is a good idea.<br>
<div class="m_-8604491652256131883HOEnZb">
<div class="m_-8604491652256131883h5"><br>
--<br>
____<br>
|| \\UTGERS, |---------------------------*<wbr>O*---------------------------<br>
||_// the State | Ryan Novosielski - <a href="mailto:novosirj@rutgers.edu" target="_blank">
novosirj@rutgers.edu</a><br>
|| \\ University | Sr. Technologist - <a href="tel:973%2F972.0922" target="_blank" value="+19739720922">
973/972.0922</a> (2x0922) ~*~ RBHS Campus<br>
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark<br>
`'<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</body>
</html>