[slurm-users] Nagios or Other Monitoring Plugins

[Ryan Novosielski]

Would be interested if anyone knows of any such scripts that already exist.

--
____
|| \\UTGERS,       |---------------------------*O*---------------------------
||_// the State     |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ     | Office of Advanced Research Computing - MSB C630, Newark
    `'

On Jan 19, 2018, at 03:44, John Hearns <hearnsj at googlemail.com<mailto:hearnsj at googlemail.com>> wrote:

Not specifically Slurm, but it can be useful to have alerts on jobs which either will never start or which are 'stalled'.
You might want to have an alert on jobs which (say) request more slots or nodes than physicall exist, so the users job will never run.
Or you can look for 'stalled' jobs where the CPU time used never increases.

On 19 January 2018 at 08:56, Marcin Stolarek <stolarek.marcin at gmail.com<mailto:stolarek.marcin at gmail.com>> wrote:
We're using icinga2 storing accounting data in influxdb for grafana dashboards. In terms of monitoring I prefere end-user functionality, so apart from services we also have a plugin that submits a jobs to cluster (to idle nodes, with a few minutes of deadline) the job simply creates files on shared filesystem effectively monitoring slurmctl, slurmd, sssd, filesystems etc.

cheers,
Marcin

2018-01-19 5:44 GMT+01:00 Ryan Novosielski <novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>>:
> On Jan 18, 2018, at 4:34 PM, Lachlan Musicman <datakid at gmail.com<mailto:datakid at gmail.com>> wrote:
>
> On 19 January 2018 at 07:29, Ryan Novosielski <novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>> wrote:
> Hi all,
>
> Looked back at the mailing list to see if there was a question about this already. There was some mention of /using/ Nagios, but no real mention of specifics. What do people monitor with Nagios? We monitor, so far, slurmctld, slurmdbd, and MySQL, but there are probably some others. Might be helpful to run “scontrol ping” for example, or similar, on our login nodes.
>
> Does anyone have any plugins they’ve written or ideas they can share? Nagios Exchange doesn’t have anything with SLURM anywhere in the name.
>
> Thanks!
>
>
> Off the top of my head the only other two that I would want explicitly would be:
>  - ntp/chrony and their respective ntpd. Nodes go offline when the timing slides too far, especially if you are using Munge.
>  - authentication system - in our case ipa/sssd. Without that, even the queued jobs will fail.
>
> We use Zabbix in house. I was under the impression that people were moving toward icingia2 over Nagios.

I wouldn’t mind moving to Icinga2 over Nagios, but really, it’s more or less a nicer version of the same thing, so I’d have the same question with Icinga2.

Thanks for the NTP/Chrony tip though — if I get only that from this thread, it will have been worth it. That’s caused us trouble more than once. We do already monitor our LDAP, but SSSD is a good idea.

--
____
|| \\UTGERS,     |---------------------------*O*---------------------------
||_// the State  |         Ryan Novosielski - novosirj at rutgers.edu<mailto:novosirj at rutgers.edu>
|| \\ University | Sr. Technologist - 973/972.0922<tel:973%2F972.0922> (2x0922) ~*~ RBHS Campus
||  \\    of NJ  | Office of Advanced Research Computing - MSB C630, Newark
     `'


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20180120/9228ad0a/attachment.html>