Dear Slurm users, Following the initial announcement below, I have been improving the robustness of the crontab job "update_down_node.sh" script in [1], mainly by using the little-known command "scontrol show slurmd" to inquire reliably the state of the slurmd service. We have been making security updates of our entire cluster multiple times per week using this method, and it also works well on GPU nodes where slurmd may take 1-2 minutes to start up. I would welcome any suggestions or questions regarding these tools. Best regards, Ole [1] https://github.com/OleHolmNielsen/Slurm_tools/tree/master/nodes#compute-node... On 10/15/26 14:14, Ole Holm Nielsen wrote:
Dear Slurm users,
Keeping Slurm compute nodes up-to-date with security updates is an important task, and since security updates are now released with a high frequency, it is desirable to make the updating process as simple and automatic as possible. The tools documented here in [1] offer a possible solution. It's enlightening to browse OS errata pages such as [2] or [3].
The design used in the present approach [1] is:
* Configure all compute nodes to execute a crontab job script at every reboot. This script will usually do nothing, except in case the node has a Slurm "State=down" where it will perform the configured updates.
* Making updates simply consists of rebooting compute node lists with the command "scontrol reboot asap nextstate=down <node-list>". Here we have employed (abused?) the Slurm "State=down" as a *trigger* telling the update script to perform some actions. Nodes with a "Reason=Node unexpectedly rebooted" are assumed to be unrelated to the present update system and will therefore be skipped.
The page [1] gives details of configuration and the optional handling of mixed-state nodes.
We have tested these tools on our 700-node cluster running RockyLinux 8 and 9 for some time now, and they work reliably in our environment when updating several times per week. I would welcome any comments or bug reports regarding these tools.
Best regards, Ole
[1] https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Fgithub.com%2FOleHolmNielsen%2FSlurm_tools%2Ftree%2Fmaster%2Fnodes%23compute-node-os-and-firmware-updates&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925091389%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MBaiEJFrKF5gn8z7cWw6U04%2BuGuMMRBG0uVw601jH2Q%3D&reserved=0
[2] RockyLinux errata: https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Ferrata.build.resf.org%2F&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925137734%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=AEOVmLeAGoNAWXAABJOuDgVU0or7NAg0iYwKW2WHXSE%3D&reserved=0
[3] AlmaLinux errata: https://eur01.safelinks.protection.outlook.com/? url=https%3A%2F%2Ferrata.almalinux.org%2F&data=05%7C02%7COle.H.Nielsen%40fysik.dtu.dk%7Cb7cd2774dbeb4e72b1a808df2ab5e8e2%7Cf251f123c9ce448e927734bb285911d9%7C0%7C0%7C639276632925188190%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=btrSuNCBAq%2FGpMcJPeYMxFl3l2TARLSLw1Mbs3nxVMM%3D&reserved=0
-- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark