Making easy and simple security updates of Slurm compute nodes
Dear Slurm users, Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3]. The design used in the present approach [1] is: * Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates. * Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped. The page [1] gives details of configuration and the optional handling of mixed-state nodes. We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-node... [2] RockyLinux errata: https://errata.build.resf.org/ [3] AlmaLinux errata: https://errata.almalinux.org/ -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
Dear Slurm users, Following the initial announcement below, I have been improving the robustness of the crontab job "update_down_node.sh" script in [1], mainly by using the little-known command "scontrol show slurmd" to inquire reliably the state of the slurmd service. We have been making security updates of our entire cluster multiple times per week using this method, and it also works well on GPU nodes where slurmd may take 1-2 minutes to start up. I would welcome any suggestions or questions regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-node... On 10/15/26 14:14, Ole Holm Nielsen wrote:
Dear Slurm users,
Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3].
The design used in the present approach [1] is:
* Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates.
* Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped.
The page [1] gives details of configuration and the optional handling of mixed-state nodes.
We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools.
Best regards, Ole
[1] https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Fgithub.com%2FOleHolmNielsen%2FSlurm_tools%2Ftree%2Fmaster%2Fnodes%23compute-node-os-and-firmware-updates&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925091389%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MBaiEJFrKF5gn8z7cWw6U04%2BuGuMMRBG0uVw601jH2Q%3D&reserved=0
[2] RockyLinux errata: https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Ferrata.build.resf.org%2F&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925137734%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AEOVmLeAGoNAWXAABJOuDgVU0or7NAg0iYwKW2WHXSE%3D&reserved=0
[3] AlmaLinux errata: https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Ferrata.almalinux.org%2F&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925188190%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=btrSuNCBAq%2FGpMcJPeYMxFl3l2TARLSLw1Mbs3nxVMM%3D&reserved=0
-- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
I have put off applying updates as our Slurm install was built from RPMs we built, selecting the options that we need - as I understand it features are included if the corresponding 'dev' libraries are found on the build system. We used the process documented here https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#build-slurm-rp... We could now use the Rocky Linux 10 RPMs (indeed the Errata keeps offering to do updates) but I have assumed that these would not include or exclude the features in the same way, so if we allowed it to update this would break Slurm. Or is this not so? It would of course be desirable to be able to apply at least all security updates promptly after they are released. William
Hi William, On 10/26/26 13:49, william--- via slurm-users wrote:
I have put off applying updates as our Slurm install was built from RPMs we built, selecting the options that we need - as I understand it features are included if the corresponding 'dev' libraries are found on the build system. We used the process documented here https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#build-slurm-rpms>>
I'm glad my Slurm Wiki pages have been of use to you :-) I'm unsure what you mean by 'dev' libraries?
We could now use the Rocky Linux 10 RPMs (indeed the Errata keeps offering to do updates) but I have assumed that these would not include or exclude the features in the same way, so if we allowed it to update this would break Slurm. Or is this not so?
Maybe I'm misunderstanding you, but my mail wasn't referring to Slurm updates, but to general RockyLinux (or other EL distributions) where kernel and many other updates are currently coming out daily!
It would of course be desirable to be able to apply at least all security updates promptly after they are released.
This is what we do. My method basically does "dnf upgrade" of all updates on compute nodes in a controlled manner and without disturbing any user jobs. Best regards, Ole
Indeed I had misunderstood you, I thought that you were applying Slurm point upgrades automatically. I think that the reason for that misunderstanding is that we use Foreman & Katello for our patching and for all our Slurm nodes (on, I think 25.04) are flagged as needing an Erratum applied that is Slurm 26.05 or the like, to fix a Slurm security bug. Every time that I do patching (which is not yet automatically scheduled) I have to be careful to de-select that Erratum, and only apply the others. Your process now makes more sense to me. We only have 22 nodes so I can manage a while more with manually draining subsets of the cluster and applying patches. This would be easier if we did not use LUKS-encrypted disks as on every reboot we still have to manually enter the encryption key, but that is quite a different problem. Thanks for clarifying that, and your Slurm Wiki pages have been very useful to me for many years now. William -----Original Message----- From: Ole Holm Nielsen via slurm-users <slurm-users@lists.schedmd.com> Sent: 26 October 2026 13:35 To: slurm-users@lists.schedmd.com Subject: [slurm-users] Re: Making easy and simple security updates of Slurm compute nodes Hi William, On 10/26/26 13:49, william--- via slurm-users wrote:
I have put off applying updates as our Slurm install was built from RPMs we built, selecting the options that we need - as I understand it features are included if the corresponding 'dev' libraries are found on the build system. We used the process documented here https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_installation/#build-sl urm-rpms>>
I'm glad my Slurm Wiki pages have been of use to you :-) I'm unsure what you mean by 'dev' libraries?
We could now use the Rocky Linux 10 RPMs (indeed the Errata keeps offering to do updates) but I have assumed that these would not include or exclude the features in the same way, so if we allowed it to update this would break Slurm. Or is this not so?
Maybe I'm misunderstanding you, but my mail wasn't referring to Slurm updates, but to general RockyLinux (or other EL distributions) where kernel and many other updates are currently coming out daily!
It would of course be desirable to be able to apply at least all security updates promptly after they are released.
This is what we do. My method basically does "dnf upgrade" of all updates on compute nodes in a controlled manner and without disturbing any user jobs. Best regards, Ole -- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
participants (2)
-
Ole Holm Nielsen -
william＠signalbox.org.uk