We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
On Tue, 2025-08-19 at 10:20 -0400, Paul Edmon via slurm-users wrote:
We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
I guess that checking for open ports with nmap should be sufficient to tell that the daemons are up and responding:
nmap -p <port> <ip-address range>
VB
I havent run this as a Node Health Check script. However it does run undel RHEL 9
https://github.com/amd/node-scraper
On Tue, 19 Aug 2025 at 15:23, Paul Edmon via slurm-users < slurm-users@lists.schedmd.com> wrote:
We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
On Tue, 2025-08-19 at 16:05 +0100, John Hearns via slurm-users wrote:
I havent run this as a Node Health Check script. However it does run undel RHEL 9
It is a python script, it should run on any system.
https://github.com/amd/node-scraper
On Tue, 19 Aug 2025 at 15:23, Paul Edmon via slurm-users slurm-users@lists.schedmd.com wrote:
We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
Hi Paul,
the dev branch of NHC is more up to date (though also 7 months stale now) and we are running this on RHEL9.6 with Slurm 24.11. Admittedly, we haven't been running it very long yet, so there might be issues we just haven't encountered yet, but in general it seems to work.
Kind regards, Frank
-- Dr. Frank Otto Principal Research Infrastructure Developer UCL Advanced Research Computing Centre Tel: 020 7679 1506 ________________________________ From: Paul Edmon via slurm-users slurm-users@lists.schedmd.com Sent: 19 August 2025 15:20 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Node Health Check Program
⚠ Caution: External sender
We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Great. I will check that out.
-Paul Edmon-
On 8/19/25 11:45 AM, Otto, Frank wrote:
Hi Paul,
the dev branch of NHC is more up to date (though also 7 months stale now) and we are running this on RHEL9.6 with Slurm 24.11. Admittedly, we haven't been running it very long yet, so there might be issues we just haven't encountered yet, but in general it seems to work.
Kind regards, Frank
-- Dr. Frank Otto Principal Research Infrastructure Developer UCL Advanced Research Computing Centre Tel: 020 7679 1506
*From:* Paul Edmon via slurm-users slurm-users@lists.schedmd.com *Sent:* 19 August 2025 15:20 *To:* slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com *Subject:* [slurm-users] Node Health Check Program ⚠ Caution: External sender
We've been using NHC (https://github.com/mej/nhc) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi Paul!
Have you by chance given the `dev` branch a try? All our production servers currently run `lbnl-nhc-1.5-0.82.gf8dc.el8.noarch` built from the `dev` branch, have been for some time now, and it's been rock solid. Our RHEL-based clusters also use this version. Our HPE/Cray Shasta clusters, including our largest (classified) clusters Crossroads, Tycho, and Venado, use a variant. (Long story short, I've merged in all my changes into a separate branch, but the reverse is not yet true.) This variant is, at present, COS/SLES-specific, but it has quite a few useful additional checks (many of them Cray-centric) contributed by other LANL folks that I haven't had a chance to upstream yet.
Well, to be fair, that's not exactly true. I could just add them in en masse — no tidying, no unit tests, no code reviews — and believe me, I've come close to doing it several times! If enough folks out there would find it useful at their site(s), I'm sure I could be persuaded. :-)
For better or worse, RHEL9 was only recently approved by the security folks here, so very few servers and clusters are running it (or derivatives like Alma or Rocky). I do have a RHEL9-based VM running the same version I noted above (for servers); the only problem I ran into so far is updating the `sshd` service check due to the fact that, with OpenSSH 8.7, even the primary daemon process (the listener) rewrites its `argv[]` data. Here's what I'm using that works:
* || check_ps_service -VS -u root -fm 'sshd: /*/sshd* -D*' sshd
I don't see any other mention of RHEL9-centric issue reports; did I miss one?
In any event, the project isn't dead, I swear! And for what it's worth, it won't be going away any time soon; LANL HPC (independently of myself) evaluated all the available options at the time, on at least 3 separate occasions, and consistently found `lbnl-nhc` to be the best choice. Since that time, it's been deployed on almost all services hosts (Quay servers, GitLab servers, OpenShift prime and worker nodes, our virtualization cluster, and numerous others) as well as all production clusters and supporting infrastructure. NHC feeds its results into Splunk (hence the recently added JSON support), and we also use things like telegraf and LDMS, but in terms of situational awareness at the OS, scheduler, and cluster hardware levels, NHC is everywhere, and we've invested quite a bit into it in terms of time, training, and ancillary efforts (like our NHC Ansible role).
I'm not able to spend 90+% of my time on NHC right now, as I was briefly able to do last year, but it is still being developed and deployed at scale.
As far as forks go, the only thing I'm aware of in that vein is work that comes from the great team over at the University of Ghent in Belgium. Their tree (github.com/hpcugent/nhc) was still undergoing development while I was stuck in legal limbo with Feynman, but I haven't checked recently to see if they've merged any of the recent features (and some pretty significant bugfixes, primarily around process management).
Hope that helps! Michael
-- Michael E. Jennings (he/him) mej@lanl.gov https://hpc.lanl.gov/ HPC Platform Integration Engineer - Platforms Design Team - HPC Design Group Ultra-Scale Research Center (USRC), 4200 W Jemez #301-25 +1 (505) 412-4151 Los Alamos National Laboratory, P.O. Box 1663, Los Alamos, NM 87545-0001
________________________________________ From: Paul Edmon via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, August 19, 2025 08:20 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [EXTERNAL] [slurm-users] Node Health Check Program We've been using NHC (https://urldefense.com/v3/__https://github.com/mej/nhc__;!!Bt8fGhp8LhKGRg!Bk... ) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thanks, that explains why no EL9 release has appeared yet. I tried out the dev branch and it works great.
NHC has been awesome to use (we've been using it for years). Thanks for maintaining it!
-Paul Edmon-
On 8/19/25 3:25 PM, Jennings, Michael E wrote:
Hi Paul!
Have you by chance given the `dev` branch a try? All our production servers currently run `lbnl-nhc-1.5-0.82.gf8dc.el8.noarch` built from the `dev` branch, have been for some time now, and it's been rock solid. Our RHEL-based clusters also use this version. Our HPE/Cray Shasta clusters, including our largest (classified) clusters Crossroads, Tycho, and Venado, use a variant. (Long story short, I've merged in all my changes into a separate branch, but the reverse is not yet true.) This variant is, at present, COS/SLES-specific, but it has quite a few useful additional checks (many of them Cray-centric) contributed by other LANL folks that I haven't had a chance to upstream yet.
Well, to be fair, that's not exactly true. I could just add them in en masse — no tidying, no unit tests, no code reviews — and believe me, I've come close to doing it several times! If enough folks out there would find it useful at their site(s), I'm sure I could be persuaded. :-)
For better or worse, RHEL9 was only recently approved by the security folks here, so very few servers and clusters are running it (or derivatives like Alma or Rocky). I do have a RHEL9-based VM running the same version I noted above (for servers); the only problem I ran into so far is updating the `sshd` service check due to the fact that, with OpenSSH 8.7, even the primary daemon process (the listener) rewrites its `argv[]` data. Here's what I'm using that works:
- || check_ps_service -VS -u root -fm 'sshd: /*/sshd* -D*' sshd
I don't see any other mention of RHEL9-centric issue reports; did I miss one?
In any event, the project isn't dead, I swear! And for what it's worth, it won't be going away any time soon; LANL HPC (independently of myself) evaluated all the available options at the time, on at least 3 separate occasions, and consistently found `lbnl-nhc` to be the best choice. Since that time, it's been deployed on almost all services hosts (Quay servers, GitLab servers, OpenShift prime and worker nodes, our virtualization cluster, and numerous others) as well as all production clusters and supporting infrastructure. NHC feeds its results into Splunk (hence the recently added JSON support), and we also use things like telegraf and LDMS, but in terms of situational awareness at the OS, scheduler, and cluster hardware levels, NHC is everywhere, and we've invested quite a bit into it in terms of time, training, and ancillary efforts (like our NHC Ansible role).
I'm not able to spend 90+% of my time on NHC right now, as I was briefly able to do last year, but it is still being developed and deployed at scale.
As far as forks go, the only thing I'm aware of in that vein is work that comes from the great team over at the University of Ghent in Belgium. Their tree (github.com/hpcugent/nhc) was still undergoing development while I was stuck in legal limbo with Feynman, but I haven't checked recently to see if they've merged any of the recent features (and some pretty significant bugfixes, primarily around process management).
Hope that helps! Michael
-- Michael E. Jennings (he/him) mej@lanl.gov https://hpc.lanl.gov/ HPC Platform Integration Engineer - Platforms Design Team - HPC Design Group Ultra-Scale Research Center (USRC), 4200 W Jemez #301-25 +1 (505) 412-4151 Los Alamos National Laboratory, P.O. Box 1663, Los Alamos, NM 87545-0001
From: Paul Edmon via slurm-users slurm-users@lists.schedmd.com Sent: Tuesday, August 19, 2025 08:20 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [EXTERNAL] [slurm-users] Node Health Check Program
We've been using NHC (https://urldefense.com/v3/__https://github.com/mej/nhc__;!!Bt8fGhp8LhKGRg!Bk... ) for years with much success. However that project hasn't had a release in 2 years and the various Issues filed indicate that there might be problems with Rocky 9 (which we are looking to upgrade to). Do people that are at EL9 use NHC? Is there a fork? Is there a different code that people use for doing node health checks?
-Paul Edmon-
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
On 8/19/25 21:25, Jennings, Michael E via slurm-users wrote:
Have you by chance given the `dev` branch a try? All our production servers currently run `lbnl-nhc-1.5-0.82.gf8dc.el8.noarch` built from the `dev` branch, have been for some time now, and it's been rock solid. Our RHEL-based clusters also use this version. Our HPE/Cray Shasta clusters, including our largest (classified) clusters Crossroads, Tycho, and Venado, use a variant. (Long story short, I've merged in all my changes into a separate branch, but the reverse is not yet true.) This variant is, at present, COS/SLES-specific, but it has quite a few useful additional checks (many of them Cray-centric) contributed by other LANL folks that I haven't had a chance to upstream yet.
Due to Michael's recommendation I wanted to try out the 'dev' branch version 1.5 of NHC and build an RPM package referred to by Michael.
Since I'm not a software developer, I had to figure out for myself the detailed building steps - perhaps trivial to some of you, and stumbling blocks to others. This is what I came up with:
$ git clone https://github.com/mej/nhc.git $ cd nhc $ git switch dev # Switch to the 'dev' branch $ git status # Check the status $ grep nhc_version configure.ac # Verify the 'dev' version m4_define([nhc_version], [1.5]) $ ./autogen.sh # Undocumented build requirement $ cd .. $ mv nhc lbnl-nhc-1.5 # Rename the source folder $ tar czf lbnl-nhc-1.5.tar.gz lbnl-nhc-1.5 $ rpmbuild -ta lbnl-nhc-1.5.tar.gz
The resulting RPM package is:
~/rpmbuild/RPMS/noarch/lbnl-nhc-1.5-0.82.gf8dc.el8.noarch.rpm
I've added those steps to my Slurm Wiki page: https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#node-health-c...
Any comments?
Thanks, Ole
Hi Ole,
this looks similar to what I've been doing for building RPMs. (It's documented for our in-house branch at [1], if anyone wants to compare.) Happy to see I'm not doing something totally stupid. :)
[1] https://github.com/UCL-ARC/nhc/blob/ucl/README.md
Thanks, Frank
-- Dr. Frank Otto Principal Research Infrastructure Developer Advanced Research Computing Centre Univesity College London, UK ________________________________ From: Ole Holm Nielsen via slurm-users slurm-users@lists.schedmd.com Sent: 22 August 2025 12:17 To: slurm-users@lists.schedmd.com slurm-users@lists.schedmd.com Subject: [slurm-users] Re: [EXTERNAL] Node Health Check Program
⚠ Caution: External sender
On 8/19/25 21:25, Jennings, Michael E via slurm-users wrote:
Have you by chance given the `dev` branch a try? All our production servers currently run `lbnl-nhc-1.5-0.82.gf8dc.el8.noarch` built from the `dev` branch, have been for some time now, and it's been rock solid. Our RHEL-based clusters also use this version. Our HPE/Cray Shasta clusters, including our largest (classified) clusters Crossroads, Tycho, and Venado, use a variant. (Long story short, I've merged in all my changes into a separate branch, but the reverse is not yet true.) This variant is, at present, COS/SLES-specific, but it has quite a few useful additional checks (many of them Cray-centric) contributed by other LANL folks that I haven't had a chance to upstream yet.
Due to Michael's recommendation I wanted to try out the 'dev' branch version 1.5 of NHC and build an RPM package referred to by Michael.
Since I'm not a software developer, I had to figure out for myself the detailed building steps - perhaps trivial to some of you, and stumbling blocks to others. This is what I came up with:
$ git clone https://github.com/mej/nhc.git $ cd nhc $ git switch dev # Switch to the 'dev' branch $ git status # Check the status $ grep nhc_version configure.ac # Verify the 'dev' version m4_define([nhc_version], [1.5]) $ ./autogen.sh # Undocumented build requirement $ cd .. $ mv nhc lbnl-nhc-1.5 # Rename the source folder $ tar czf lbnl-nhc-1.5.tar.gz lbnl-nhc-1.5 $ rpmbuild -ta lbnl-nhc-1.5.tar.gz
The resulting RPM package is:
~/rpmbuild/RPMS/noarch/lbnl-nhc-1.5-0.82.gf8dc.el8.noarch.rpm
I've added those steps to my Slurm Wiki page: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.fysik...https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#node-health-check
Any comments?
Thanks, Ole
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
On 8/19/25 21:25, Jennings, Michael E via slurm-users wrote:
Have you by chance given the `dev` branch a try? All our production servers currently run `lbnl-nhc-1.5-0.82.gf8dc.el8.noarch` built from the `dev` branch, have been for some time now, and it's been rock solid. Our RHEL-based clusters also use this version. Our HPE/Cray Shasta clusters, including our largest (classified) clusters Crossroads, Tycho, and Venado, use a variant. (Long story short, I've merged in all my changes into a separate branch, but the reverse is not yet true.) This variant is, at present, COS/SLES-specific, but it has quite a few useful additional checks (many of them Cray-centric) contributed by other LANL folks that I haven't had a chance to upstream yet.
I've built and tested the lbnl-nhc-1.5-0.82.gf8dc.el8.noarch RPM which is built as described in my previous mail. Unfortunately LBNL NHC version 1.5-0.82.gf8dc fails to recognize NHC_RM=slurm in nhc.conf. We get this error:
/usr/libexec/nhc/node-mark-online: Unsupported RM detected in /usr/libexec/nhc/node-mark-online: ""
I created a new issue https://github.com/mej/nhc/issues/165
We need this issue to be fixed before we can deploy NHC 1.5 :-(
Any suggestions?
Thanks, Ole
-
Jennings, Michael E -
John Hearns -
Ole Holm Nielsen -
Otto, Frank -
Paul Edmon -
Valerio Bellizzomi