September 2026 - slurm-users - lists.schedmd.com

Timed-out jobs were 6.6% of jobs but ~45% of job energy on two DOE clusters
by James Jardine 22 Sep '26

22 Sep '26

Hello, I've been working through job-scheduler records from two DOE clusters (Eagle and Kestrel) and one result seems worth putting in front of this list. Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of total job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing. The part I didn't expect is how predictable they are before they start. Four summary statistics (mean, standard deviation, range, mean absolute successive difference) over a user's previous 24 Elapsed/Timelimit ratios, plain logistic regression, using nothing at all about the job being predicted. Forward-chained by month, never trained on anything later than the month being scored. The prediction figures are Kestrel only: about 6.8M job windows over 22 monthly splits. At an 80th-percentile threshold, pooled across splits, 95.4% of timed-out jobs were flagged, at the cost of flagging 16.9% of jobs that would have finished fine. Per split that is 93.2% +/- 6.3 and 21.3% +/- 12.5. The false-flag spread is wide, and it is the number that decides whether this is usable at a given site, so I would look at that before the headline. The model is deliberately boring, and that is the point. What took the work was the measurement around it: forward-chaining by month so nothing is ever trained on the future, a label-shuffle control that has to land at chance before any number is printed, and a refusal path so it prints nothing at all when the data won't carry a conclusion. 868 lines total, so you can audit the lot in an afternoon. I packaged it so you can run it against your own sacct export: git clone https://github.com/JJardine919/lattice24-assess cd lattice24-assess && python3 -m venv .venv && .venv/bin/pip install -e . sacct --allocations --parsable2 --starttime=$(date -d '12 months ago' +%F) \ -o User,End,Timelimit,Elapsed,State,ConsumedEnergyRaw > export.psv .venv/bin/lattice24-assess export.psv --out ./report If your site already has numpy and scikit-learn available, skip the venv and run python3 -m lattice24_assess.cli export.psv --out ./report from the clone. It runs entirely on the login node. There are no network calls in it, and the source is short enough to confirm that yourself in a couple of minutes. User IDs are hashed with a per-run salt; job names, working directories and command lines are dropped at parse time and never read. It refuses rather than degrades: under 6 months of data, under 50k windows, under 200 timeout events, or if the label-shuffle control doesn't land near chance, it explains why and prints no headline number. Two things it deliberately will not tell you. It reports energy AT STAKE, not energy saved -- a flag avoids nothing until somebody resizes, fixes or declines the job, and that multiplier belongs to your site, not to me. And it covers TIMEOUT only; CANCELLED and FAILED are excluded, so your recoverable waste is probably larger than what it reports. Repo, Apache-2.0: https://github.com/JJardine919/lattice24-assess Method and results: https://doi.org/10.5281/zenodo.21911351 I'd genuinely like to know whether the pattern holds elsewhere. If your number comes out much lower, that is a useful result and I'd like to hear it. Jim Jardine Lattice24

2 3

Suspending jobs when preempted
by pie＠lem.com 13 Sep '26

13 Sep '26

I am new to Slurm, sorry if my question has been asked before, I could only find some older discussions about this. I am currently setting up a job scheduler in our company. Main bottleneck is licenses, so a license scheduler including preemption should be included. I have noticed this in the documentation: "Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set." Which means I cannot set the PreemptMode to SUSPEND, is that correct? Is that something that will ever be implemented? I have created tres licences and implemented a polling mechanism that keeps the slurm tres licenses in sync with the real licenses. Our application automatically releases licenses when jobs get a a TSTP signal. So, all I want is the Slurm scheduler to suspend low prio jobs when needed, send a TSTP signal and let the book-keeping be done by my polling script. Is this in any way possible?

2 3

reserving TRES(gpus) fails via nodes busy
by Äkäslompolo Simppa 04 Sep '26

04 Sep '26

Hi! I've been trying to reserve a couple of GPUs permanently for a specific user, but I keep getting "Requested nodes are busy" (slurm 26.05.3) Perhaps someone could give ideas what debug flags to enable or other ways to fish out more information why slurm says the nodes are busy? Here is an example case, but I've not figured out the systematics: (And flags=ignore_jobs does not make a difference) # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g,gres/gpu:h200:4 corecnt=48 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g corecnt=48 Reservation created: llmgw_permanent # Here is another case: # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:4 flags=ignore_jobs, nodes=gpu49 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:0 flags=ignore_jobs, nodes=gpu49 Reservation created: llmgw_permanent # Thank you! -- - Simppa - Mr. Simppa Äkäslompolo High performance computing specialist Doctor of Science (Tech.) Aalto Scientific Computing School of Science, Aalto University, Finland +358-50-5311327 https://scicomp.aalto.fi/

2 1

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available (CVE-2026-65107 et seq)
by Tim Wickberg 02 Sep '26

02 Sep '26

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available and include a number of security fixes (CVE-2026-65107 et seq). Slurm support customers were informed on August 19th and provided patches on request. Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases The following excerpt from the changelogs summarizes the issues and their associated CVE ids: * Fix possible slurmstepd crash on invalid step socket requests. CVE-2026-65168. * Fix sbcast shared objects skipping credential verification. CVE-2026-65107. * Fix possible slurmd crash on invalid sbcast filenames. CVE-2026-65107. * Fix a slurmstepd stack overflow when a job environment contains an oversized SPANK option variable. CVE-2026-65108. * Fix slurmstepd removing files outside the container spool directory when cleaning up an OCI container. CVE-2026-65109. * Fix slurmstepd leaving OCI container spool directories behind when ContainerPath contains a task id pattern. CVE-2026-65109. * Fix heap over-read when unpacking a malformed forward data RPC in slurmd. CVE-2026-65138. * Fix a slurmd crash when handling a malformed forward data RPC with a missing socket address. CVE-2026-65138. * Reject cluster names containing characters that are unsafe to use in an accounting database query. CVE-2026-65139. * Reject unsafe cluster names on the accounting usage, add and runaway job paths. CVE-2026-65139. * Reject unsafe cluster names when opening a connection to the slurmdbd. CVE-2026-65139. * Reject non-numeric id values in accounting database queries. CVE-2026-65139. * Fix a privilege escalation where an operator could alter Administrator accounts through the accounting database. CVE-2026-65140. * Fix node count of a job step using arbitrary distribution. CVE-2026-65165. * Reject a hostlist function in the node list of a job step using arbitrary distribution. CVE-2026-65165. * Reject a job step whose arbitrary node list disagrees with its node count. CVE-2026-65165. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md

1 0

Any update versions of SLURM soon?
by Grigory Shamov 02 Sep '26

02 Sep '26

HI All, I have heard that a minor SLURM update is coming. Just wondering if the public Githib repo of SchedMD/slurm still gets updated releases? -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

2 1