- slurm-users - lists.schedmd.com

How Should I Size CPU and RAM for New Slurm Compute Nodes?
by wesleycolton238＠gmail.com 23 Sep '26

23 Sep '26

We’re planning to add a few new compute nodes to a small Slurm cluster, and I’m trying to work out a sensible CPU-to-memory balance before choosing the hardware. My current thought is to look at historical job data first, especially allocated CPUs, requested memory, MaxRSS, runtime, and how often jobs are waiting because of CPU versus memory availability. That seems more useful than simply buying the highest core-count CPUs available. For the new nodes, I’m mainly trying to decide: Whether fewer higher-clocked cores or more lower-clocked cores would give us better overall throughput How much RAM per physical core makes sense based on actual job usage Whether it is better to standardize all nodes or have a separate high-memory partition How much physical RAM should be left outside Slurm's RealMemory value for the OS and system processes Whether NUMA layout and memory bandwidth should influence the CPU choice as much as core count I’ve also been comparing different rack-server configurations here to get an idea of available Xeon, ECC memory, and storage combinations: https://ormsystems.com/servers-price.html For those managing production Slurm clusters, do you normally size new nodes from something like the 80th/90th percentile of historical memory-per-core usage, or do you use a different method? I’d be interested to hear what metrics you’ve found most useful before purchasing additional compute nodes.

3 2

How do I get the information available from `scontrol show job` in a format I can safely parse?
by Adam Novak 23 Sep '26

23 Sep '26

It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output. This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another. Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface? To illustrate the problem, I ran: sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")" And I found this in my `scontrol show job` output: JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment= JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power= -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

6 7

Timed-out jobs were 6.6% of jobs but ~45% of job energy on two DOE clusters
by James Jardine 22 Sep '26

22 Sep '26

Hello, I've been working through job-scheduler records from two DOE clusters (Eagle and Kestrel) and one result seems worth putting in front of this list. Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of total job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing. The part I didn't expect is how predictable they are before they start. Four summary statistics (mean, standard deviation, range, mean absolute successive difference) over a user's previous 24 Elapsed/Timelimit ratios, plain logistic regression, using nothing at all about the job being predicted. Forward-chained by month, never trained on anything later than the month being scored. The prediction figures are Kestrel only: about 6.8M job windows over 22 monthly splits. At an 80th-percentile threshold, pooled across splits, 95.4% of timed-out jobs were flagged, at the cost of flagging 16.9% of jobs that would have finished fine. Per split that is 93.2% +/- 6.3 and 21.3% +/- 12.5. The false-flag spread is wide, and it is the number that decides whether this is usable at a given site, so I would look at that before the headline. The model is deliberately boring, and that is the point. What took the work was the measurement around it: forward-chaining by month so nothing is ever trained on the future, a label-shuffle control that has to land at chance before any number is printed, and a refusal path so it prints nothing at all when the data won't carry a conclusion. 868 lines total, so you can audit the lot in an afternoon. I packaged it so you can run it against your own sacct export: git clone https://github.com/JJardine919/lattice24-assess cd lattice24-assess && python3 -m venv .venv && .venv/bin/pip install -e . sacct --allocations --parsable2 --starttime=$(date -d '12 months ago' +%F) \ -o User,End,Timelimit,Elapsed,State,ConsumedEnergyRaw > export.psv .venv/bin/lattice24-assess export.psv --out ./report If your site already has numpy and scikit-learn available, skip the venv and run python3 -m lattice24_assess.cli export.psv --out ./report from the clone. It runs entirely on the login node. There are no network calls in it, and the source is short enough to confirm that yourself in a couple of minutes. User IDs are hashed with a per-run salt; job names, working directories and command lines are dropped at parse time and never read. It refuses rather than degrades: under 6 months of data, under 50k windows, under 200 timeout events, or if the label-shuffle control doesn't land near chance, it explains why and prints no headline number. Two things it deliberately will not tell you. It reports energy AT STAKE, not energy saved -- a flag avoids nothing until somebody resizes, fixes or declines the job, and that multiplier belongs to your site, not to me. And it covers TIMEOUT only; CANCELLED and FAILED are excluded, so your recoverable waste is probably larger than what it reports. Repo, Apache-2.0: https://github.com/JJardine919/lattice24-assess Method and results: https://doi.org/10.5281/zenodo.21911351 I'd genuinely like to know whether the pattern holds elsewhere. If your number comes out much lower, that is a useful result and I'd like to hear it. Jim Jardine Lattice24

2 3

Suspending jobs when preempted
by pie＠lem.com 13 Sep '26

13 Sep '26

I am new to Slurm, sorry if my question has been asked before, I could only find some older discussions about this. I am currently setting up a job scheduler in our company. Main bottleneck is licenses, so a license scheduler including preemption should be included. I have noticed this in the documentation: "Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set." Which means I cannot set the PreemptMode to SUSPEND, is that correct? Is that something that will ever be implemented? I have created tres licences and implemented a polling mechanism that keeps the slurm tres licenses in sync with the real licenses. Our application automatically releases licenses when jobs get a a TSTP signal. So, all I want is the Slurm scheduler to suspend low prio jobs when needed, send a TSTP signal and let the book-keeping be done by my polling script. Is this in any way possible?

2 3

reserving TRES(gpus) fails via nodes busy
by Äkäslompolo Simppa 04 Sep '26

04 Sep '26

Hi! I've been trying to reserve a couple of GPUs permanently for a specific user, but I keep getting "Requested nodes are busy" (slurm 26.05.3) Perhaps someone could give ideas what debug flags to enable or other ways to fish out more information why slurm says the nodes are busy? Here is an example case, but I've not figured out the systematics: (And flags=ignore_jobs does not make a difference) # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g,gres/gpu:h200:4 corecnt=48 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g corecnt=48 Reservation created: llmgw_permanent # Here is another case: # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:4 flags=ignore_jobs, nodes=gpu49 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:0 flags=ignore_jobs, nodes=gpu49 Reservation created: llmgw_permanent # Thank you! -- - Simppa - Mr. Simppa Äkäslompolo High performance computing specialist Doctor of Science (Tech.) Aalto Scientific Computing School of Science, Aalto University, Finland +358-50-5311327 https://scicomp.aalto.fi/

2 1

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available (CVE-2026-65107 et seq)
by Tim Wickberg 02 Sep '26

02 Sep '26

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available and include a number of security fixes (CVE-2026-65107 et seq). Slurm support customers were informed on August 19th and provided patches on request. Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases The following excerpt from the changelogs summarizes the issues and their associated CVE ids: * Fix possible slurmstepd crash on invalid step socket requests. CVE-2026-65168. * Fix sbcast shared objects skipping credential verification. CVE-2026-65107. * Fix possible slurmd crash on invalid sbcast filenames. CVE-2026-65107. * Fix a slurmstepd stack overflow when a job environment contains an oversized SPANK option variable. CVE-2026-65108. * Fix slurmstepd removing files outside the container spool directory when cleaning up an OCI container. CVE-2026-65109. * Fix slurmstepd leaving OCI container spool directories behind when ContainerPath contains a task id pattern. CVE-2026-65109. * Fix heap over-read when unpacking a malformed forward data RPC in slurmd. CVE-2026-65138. * Fix a slurmd crash when handling a malformed forward data RPC with a missing socket address. CVE-2026-65138. * Reject cluster names containing characters that are unsafe to use in an accounting database query. CVE-2026-65139. * Reject unsafe cluster names on the accounting usage, add and runaway job paths. CVE-2026-65139. * Reject unsafe cluster names when opening a connection to the slurmdbd. CVE-2026-65139. * Reject non-numeric id values in accounting database queries. CVE-2026-65139. * Fix a privilege escalation where an operator could alter Administrator accounts through the accounting database. CVE-2026-65140. * Fix node count of a job step using arbitrary distribution. CVE-2026-65165. * Reject a hostlist function in the node list of a job step using arbitrary distribution. CVE-2026-65165. * Reject a job step whose arbitrary node list disagrees with its node count. CVE-2026-65165. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md

1 0

Any update versions of SLURM soon?
by Grigory Shamov 02 Sep '26

02 Sep '26

HI All, I have heard that a minor SLURM update is coming. Just wondering if the public Githib repo of SchedMD/slurm still gets updated releases? -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

2 1

HPC Client GUI: desktop client for SSH and Slurm workflows — feedback welcome
by Mehmed Sinan KömeK 28 Aug '26

28 Aug '26

Hello everyone, I would like to introduce HPC Client GUI, a desktop application I have been developing to make everyday work with SSH-accessible Slurm clusters easier. The project grew out of my own research and CFD workflows on Türkiye’s TRUBA infrastructure. I wanted to reduce the need to switch between a terminal, an SFTP client, scheduler commands, and job-output files. HPC Client GUI brings these tasks together: - SSH connections and reusable connection profiles - Remote file browsing, SFTP uploads/downloads, and transfer management - Slurm job submission, queue monitoring, accounting information, and cancellation - Job output and error-file inspection - Integrated terminal and CLI workflows - Optional X11 forwarding - Downloadable cluster profiles, job templates, and lint rule packs, including ANSYS Fluent journal checks The application runs locally on Windows, macOS, and Linux. It uses SSH, SFTP, and standard Slurm commands such as sbatch, squeue, sacct, and scancel; it does not require a server-side agent, web portal, or slurmrestd deployment. Repository, screenshots, documentation, and downloads: https://github.com/mskomek/hpc-client-gui Although TRUBA was the original use case, the project is intended for other SSH + Slurm environments. Compatibility with individual clusters depends on their authentication methods, scheduler configuration, and site policies. This is an independent project, not an official TRUBA or SchedMD tool. I would appreciate feedback from both researchers and cluster administrators, especially on: - Compatibility with different Slurm configurations and output formats - SSH authentication, MFA, and jump-host requirements - Appropriate polling frequency and login-node load - Site policies or safeguards a desktop client should respect - Missing features or documentation that would prevent adoption If anyone is willing to try it on another cluster, the repository includes a read-only compatibility validation guide. Please do not share credentials, confidential hostnames, or private configuration in public reports. For transparency, the project is source-available under the PolyForm Noncommercial License 1.0.0; commercial use requires a separate license. The current macOS packages are unsigned and not notarized, as disclosed in the release notes. Thank you for your time. Suggestions, compatibility reports, and critical feedback are all welcome. Best regards, Mehmed Sinan Kömek

1 0

cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup
by Xaver Stiensmeier 24 Aug '26

24 Aug '26

Dear slurm-user list, currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed"). Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue: [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18 My *cgroup.conf*: CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100 From *slurm.conf*: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup *Slurm Version: *slurm 24.11.3 As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas! Best, Xaver

3 4

Slurm version 26.05.3 is now available
by Tim McMullan 13 Aug '26

13 Aug '26

We are pleased to announce the availability of Slurm version 26.05.3. This release fixes some mild to moderate severity issues and crashes, including srun potentially hanging during step complete due to a race and emitting a new warning when innodb_snapshot_isolation is set to ON. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Tim McMullan Senior System Software Engineer NVIDIA

1 0