Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750 I hope that the fix will be included in 26.05.5. We have had to disable power saving until this issue has been resolved. Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
Hi, Could someone share the (buggy) traces of the sinfo/squeue commands , in this format? https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a1d/q... I would help: A fellow in Boston would make the detection of that node state more visible in said tool. BR, F. On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users < slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750
I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi, @Ole thank you for reporting the bug to SchedMD. I hope the fix is shipped soon. On the one hand I can understand that this not included in 26.05.4 as we indeed upgrade due the CVEs. On the other hand we have now a buggy release. @Fotis The format doesn't look very special. As written in the original mail the state in sinfo should be written as "unk". Please understand that I don't want to trigger the bug again just for "fun". We have a automatically running health script to handle this kind of issues, but this to wake-up all idle~ nodes. If I have to run reconfigure before the bug is fixed I will try to accommodate your request. Best, Stefan Am Dienstag, 29. September 2026, 15:30:37 Mitteleuropäische Sommerzeit schrieb Fotis Georgatos via slurm-users:
Hi,
Could someone share the (buggy) traces of the sinfo/squeue commands , in this format? https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a1d/q topconf.yaml#L31
I would help: A fellow in Boston would make the detection of that node state more visible in said tool.
BR, F.
On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users <
slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750
I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
This is a really insightful contribution to the thread. When looking at research trends across evolutionary biology and behavioral sciences, tracking broader analytical frameworks and industry datasets—similar to the market intelligence methods used by Roots Analysis—often reveals valuable interdisciplinary patterns. Thank you for sharing these updates with the group! https://www.rootsanalysis.com/industrial-automation-market
@Stefan fyi. no need to harvest the unk state, it got treated here: https://github.com/qtop/qtop/blob/14240dbc8325f07ced65ba77158ff8997bdcf99c/q... On Wed, 30 Sept 2026 at 09:55, Stefan Staeglich via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hi,
@Ole thank you for reporting the bug to SchedMD.
I hope the fix is shipped soon. On the one hand I can understand that this not included in 26.05.4 as we indeed upgrade due the CVEs. On the other hand we have now a buggy release.
@Fotis The format doesn't look very special. As written in the original mail the state in sinfo should be written as "unk".
Please understand that I don't want to trigger the bug again just for "fun". We have a automatically running health script to handle this kind of issues, but this to wake-up all idle~ nodes. If I have to run reconfigure before the bug is fixed I will try to accommodate your request.
Best, Stefan
Am Dienstag, 29. September 2026, 15:30:37 Mitteleuropäische Sommerzeit schrieb Fotis Georgatos via slurm-users:
Hi,
Could someone share the (buggy) traces of the sinfo/squeue commands , in this format?
https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a1d/q
topconf.yaml#L31
I would help: A fellow in Boston would make the detection of that node state more visible in said tool.
BR, F.
On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users <
slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750
I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Thank you. In the meantime, we realized that the error occurred regularly. But it was automatically caught by our scripts, so we didn't notice it right away. That's why we decided to ship the patch ourselves. Since 26.05 is still new and probably not many sites have switched to it yet, I don't understand why such an important bug fix was held back. This feature, which I believe is increasingly important these days, is therefore currently broken in 26.05 stable. Best, Stefan Am Dienstag, 13. Oktober 2026, 13:00:00 Mitteleuropäische Sommerzeit schrieb Fotis Georgatos:
@Stefan
fyi. no need to harvest the unk state, it got treated here: https://github.com/qtop/qtop/blob/14240dbc8325f07ced65ba77158ff8997bdcf99c/q top_py/plugins/slurm.py#L177
On Wed, 30 Sept 2026 at 09:55, Stefan Staeglich via slurm-users <
slurm-users@lists.schedmd.com> wrote:
Hi,
@Ole thank you for reporting the bug to SchedMD.
I hope the fix is shipped soon. On the one hand I can understand that this not included in 26.05.4 as we indeed upgrade due the CVEs. On the other hand we have now a buggy release.
@Fotis The format doesn't look very special. As written in the original mail the state in sinfo should be written as "unk".
Please understand that I don't want to trigger the bug again just for "fun". We have a automatically running health script to handle this kind of issues, but this to wake-up all idle~ nodes. If I have to run reconfigure before the bug is fixed I will try to accommodate your request.
Best, Stefan
Am Dienstag, 29. September 2026, 15:30:37 Mitteleuropäische Sommerzeit schrieb
Fotis Georgatos via slurm-users:
Hi,
Could someone share the (buggy) traces of the sinfo/squeue commands , in this format?
https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a1d /q>
topconf.yaml#L31
I would help: A fellow in Boston would make the detection of that node state more visible in said tool.
BR, F.
On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users <
slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running
scontrol
reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750
I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
-- Albert-Ludwigs-Universität Freiburg Institut für Informatik Professur für Maschinelles Lernen Stefan Stäglich System-Administrator T +49 761 203-8223 staeglis@informatik.uni-freiburg.de https://ml.informatik.uni-freiburg.de Georges-Köhler-Allee 52 D-79110 Freiburg
Hi Stefan, We have been running 26.05 since 2 months and we're very happy with it. Except for the mentioned regression, which has now been fixed in 26.05.5: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md
slurmctld - Fix regression from 26.05 where powered down nodes configured with the default State=UNKNOWN would be placed into UNKNOWN+POWERED_DOWN or UNKNOWN+POWERING_DOWN on reconfigure or restart, where they would need to be manually powered back up.
So we should be able to run with power saving again soon. Best regards, Ole On 10/13/26 13:18, Stefan Staeglich via slurm-users wrote:
Thank you.
In the meantime, we realized that the error occurred regularly. But it was automatically caught by our scripts, so we didn't notice it right away. That's why we decided to ship the patch ourselves.
Since 26.05 is still new and probably not many sites have switched to it yet, I don't understand why such an important bug fix was held back. This feature, which I believe is increasingly important these days, is therefore currently broken in 26.05 stable.
Best, Stefan
Am Dienstag, 13. Oktober 2026, 13:00:00 Mitteleuropäische Sommerzeit schrieb Fotis Georgatos:
@Stefan
fyi. no need to harvest the unk state, it got treated here: https://github.com/qtop/qtop/blob/14240dbc8325f07ced65ba77158ff8997bdcf99c/q>>> top_py/plugins/slurm.py#L177
On Wed, 30 Sept 2026 at 09:55, Stefan Staeglich via slurm-users <
slurm-users@lists.schedmd.com> wrote:
Hi,
@Ole thank you for reporting the bug to SchedMD.
I hope the fix is shipped soon. On the one hand I can understand that this not included in 26.05.4 as we indeed upgrade due the CVEs. On the other hand we have now a buggy release.
@Fotis The format doesn't look very special. As written in the original mail the state in sinfo should be written as "unk".
Please understand that I don't want to trigger the bug again just for "fun". We have a automatically running health script to handle this kind of issues, but this to wake-up all idle~ nodes. If I have to run reconfigure before the bug is fixed I will try to accommodate your request.
Best, Stefan
Am Dienstag, 29. September 2026, 15:30:37 Mitteleuropäische Sommerzeit schrieb
Fotis Georgatos via slurm-users:
Hi,
Could someone share the (buggy) traces of the sinfo/squeue commands , in this format?
https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a1d>>>> /q>
topconf.yaml#L31
I would help: A fellow in Boston would make the detection of that node state more visible in said tool.
BR, F.
On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users <
slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running
scontrol
reconfigure. This state is not cleaned by Slurm itself.
Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750>>>>>> I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
Hi Ole, thank you for the notification. I expected this but don't wanted to wait even longer. But I'm happy that this is finally fixed for everyone. But my point was: - Sites normally wait before upgrading to the latest release - When there a CVEs to be fixed sites still running an unsupported release (like us) want probably to upgrade to the latest release branch as possible (like us) - A bug is a fact, introducing a new bug by trying to fix it is a hypothesis - This bug breaks the power saving feature in a serious manner - So the fix should should have been part of 26.05.4. Or 26.05.5 should have been released earlier A bug is a pain. Waiting for the the finished bug fix to be released is even more pain. Best, Stefan Am Mittwoch, 14. Oktober 2026, 11:04:00 Mitteleuropäische Sommerzeit schrieb Ole Holm Nielsen via slurm-users:
Hi Stefan,
We have been running 26.05 since 2 months and we're very happy with it. Except for the mentioned regression, which has now been fixed in 26.05.5:
https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md
slurmctld - Fix regression from 26.05 where powered down nodes configured with the default State=UNKNOWN would be placed into UNKNOWN+POWERED_DOWN or UNKNOWN+POWERING_DOWN on reconfigure or restart, where they would need to be manually powered back up.
So we should be able to run with power saving again soon.
Best regards, Ole
On 10/13/26 13:18, Stefan Staeglich via slurm-users wrote:
Thank you.
In the meantime, we realized that the error occurred regularly. But it was automatically caught by our scripts, so we didn't notice it right away. That's why we decided to ship the patch ourselves.
Since 26.05 is still new and probably not many sites have switched to it yet, I don't understand why such an important bug fix was held back. This feature, which I believe is increasingly important these days, is therefore currently broken in 26.05 stable.
Best, Stefan
Am Dienstag, 13. Oktober 2026, 13:00:00 Mitteleuropäische Sommerzeit schrieb Fotis Georgatos:
@Stefan
fyi. no need to harvest the unk state, it got treated here: https://github.com/qtop/qtop/blob/14240dbc8325f07ced65ba77158ff8997bdcf99 c/q>>> top_py/plugins/slurm.py#L177
On Wed, 30 Sept 2026 at 09:55, Stefan Staeglich via slurm-users <
slurm-users@lists.schedmd.com> wrote:
Hi,
@Ole thank you for reporting the bug to SchedMD.
I hope the fix is shipped soon. On the one hand I can understand that this not included in 26.05.4 as we indeed upgrade due the CVEs. On the other hand we have now a buggy release.
@Fotis The format doesn't look very special. As written in the original mail the state in sinfo should be written as "unk".
Please understand that I don't want to trigger the bug again just for "fun". We have a automatically running health script to handle this kind of issues, but this to wake-up all idle~ nodes. If I have to run reconfigure before the bug is fixed I will try to accommodate your request.
Best, Stefan
Am Dienstag, 29. September 2026, 15:30:37 Mitteleuropäische Sommerzeit schrieb
Fotis Georgatos via slurm-users:
Hi,
Could someone share the (buggy) traces of the sinfo/squeue commands , in this format?
https://github.com/qtop/qtop/blob/c89c85425197a63c40c58d63379e93ffbfbf9a 1d>>>> /q>
topconf.yaml#L31
I would help: A fellow in Boston would make the detection of that node state more visible in said tool.
BR, F.
On Tue, 29 Sep 2026 at 14:58, Ole Holm Nielsen via slurm-users <
slurm-users@lists.schedmd.com> wrote:
On 9/29/26 13:48, Stefan Staeglich via slurm-users wrote:
> since we updated our site from Slurm 24.11 to Slurm 26.05.4 the > idle~ > (powersaved) nodes goes to state unk (unknown) whenever running
scontrol
> reconfigure. This state is not cleaned by Slurm itself. > > > > Is this a known issue of Slurm 26.05?
Yes it is! We were hit by this regression in 26.05.4, see the ticket https://support.schedmd.com/show_bug.cgi?id=25750>>>>>> I hope that the fix will be included in 26.05.5.
We have had to disable power saving until this issue has been resolved.
Best regards, Ole -- Ole Holm Nielsen PhD, Senior HPC Officer Department of Physics, Technical University of Denmark
participants (4)
-
abhaybhardwaj282＠gmail.com -
Fotis Georgatos -
Ole Holm Nielsen -
Stefan Staeglich