May 2026 - slurm-users - lists.schedmd.com

Slurm release candidate version 26.05.0rc1 is available for testing
by Tim Wickberg 07 May '26

07 May '26

We are pleased to announce the availability of Slurm release candidate 26.05.0rc1. To highlight some new features coming in 26.05: - New "srun --async" step mode that submits step processes to stepmgr to queue and eventually launch. This avoids issues with keeping a large number of srun processes backgrounded to queue up future step-based workflows. (This was previously described as "mini-batch" in the Slurm roadmap presentations.) - New topology/ring and topology/torus3d topology plugins. These implement a single-dimensional and three-dimensional topology respectively. - Slurm REST API - Add support for creating/updating/removing partitions, and for viewing the active slurmctld and slurmdbd configurations. - Dynamic Memory Resizing. A job can now release memory (and have the cgroup limits updated) using 'scontrol update' while running. A new "sbatch --mem-update=<margin>@<delay>" option can also automatically reduce the memory limit to the current usage plus a given margin percentage after a specified time. - Add topology-based sorting for node ranks when using dynamic nodes with the topology plugins. This is also available generically for topology/flat (no topology) with a new alpha_step_rank option. - Add an optimized single-node path through the scheduling logic for increased performance. - Expanded the openmetrics (Prometheus) nodes, partitions, and jobs endpoints with gpu allocation statistics. This is the first release candidate of the upcoming 26.05 release series, and represents the end of development for this release, and a finalization of the RPC and state file formats. If any issues are identified with this release candidate, please report them through https://support.schedmd.com against the 26.05.x version and we will address them before the first production 26.05.0 release is made. Please note that the release candidates are not intended for production use. A preview of the updated documentation can be found at https://slurm.schedmd.com/archive/slurm-master/ . Slurm can be downloaded from https://github.com/SchedMD/slurm/releases. The changelog for 26.05 can be found here: https://github.com/SchedMD/slurm/blob/master/CHANGELOG/slurm-26.05.md -- Tim Wickberg Director, System Software, Slurm + Slinky, NVIDIA

1 0

cgroup/v2 regression in 25.11.2 — memory.peak read disabled after first task
by Emyr James 06 May '26

06 May '26

Dear all, Heads-up for anyone running 25.11.2 with cgroup/v2 + jobacct_gather/cgroup: there's a regression in the recent series that effectively disables memory.peak accounting for every task after the first one per slurmstepd. The commit is f2d05c08e6 ("cgroup/v2 - Remove static qualifier from variable", ticket 23646, cherry-picked from e1835a1602). It dropped static from memory_peak_interface in cgroup_p_task_get_acct_data while leaving interfaces_checked static. The function now looks like: bool memory_peak_interface = false; static bool interfaces_checked = false; ... if (!interfaces_checked) { memory_peak_interface = cgroup_p_has_feature(CG_MEMCG_PEAK); interfaces_checked = true; } ... if (memory_peak_interface) { common_cgroup_get_param(..., "memory.peak", ...); } interfaces_checked persists across calls; memory_peak_interface no longer does. So the feature-probe block runs exactly once. From the second call onward memory_peak_interface is a fresh stack-local set to false, and the memory.peak read in the second if is skipped. stats->memory_peak ends up INFINITE64 for every task except the first one ever sampled. User-visible effect: MaxRSS in sacct stops reflecting the kernel-tracked peak and falls back to polling memory.current — which, at typical jobacct_gather intervals, routinely misses short-lived peaks. Which is of course exactly the problem memory.peak was added to solve (see the 2024 thread that prompted the original implementation: https://groups.google.com/g/slurm-users/c/BX1_4bztrJA) One-line fix is to put static back on memory_peak_interface. Or, arguably more robust, drop the interfaces_checked cache entirely and run the feature probe on every call — the cost is one access()-equivalent per accounting poll, basically free. he original v2 implementation in ba34862a40 was great, thanks for that — this looks like a small static-qualifier slip on the way through a cleanup. Cheers, Emyr James Head of Scientific IT CRG - Centre for Genomic Regulation

1 1

slurmdbd and slurmctld prevent alma9 login
by Pols, Maarten 06 May '26

06 May '26

Dear Community, Our Slurm cluster has been running without any issues for several months on version 25.05.3. Last Friday, we experienced a power outage which required us to restart the server. After the restart, we were unable to log in to the master node. Eventually, we managed to access the system via a safe mode workaround. Through a process of elimination, we identified the slurmdbd and slurmctld services as the root cause of the issue. Would you happen to have any idea what might have caused this behavior? We have since upgraded to version 25.11.5, which appears to be running smoothly. However, we would still like to understand the underlying cause of the problem. Thank you in advance for your help. Kind regards, Maarten

3 3

Jobs canceling when nodes become unreachable – need guidance
by Pharthiphan Asokan 05 May '26

05 May '26

Hi, We’re seeing an issue where jobs submitted via salloc are automatically cancelled when a compute node becomes temporarily unreachable. Our goal is to keep jobs pending or requeued instead of being cancelled outright when a node drops offline briefly Slurm sometimes cancels the job rather than requeuing it when the node is marked DOWN/ DRAIN/ DRAINING*. Is there a recommended configuration or additional parameter that ensures jobs remain pending/requeued until the node returns, rather than being cancelled? Any insights or examples from similar setups would be greatly appreciated. Regards, Pharthiphan

3 2