- slurm-users - lists.schedmd.com

Slurm 26.05.4: idle~ nodes go to unknown after reconfigure
by Stefan Staeglich 12 Oct '26

12 Oct '26

Hi, since we updated our site from Slurm 24.11 to Slurm 26.05.4 the idle~ (powersaved) nodes goes to state unk (unknown) whenever running scontrol reconfigure. This state is not cleaned by Slurm itself. Is this a known issue of Slurm 26.05? Best, Stefan

4 4

SlurmDB Duplicate
by Paul Edmon 28 Sep '26

28 Sep '26

Currently we have a slurmdbd that is our main production, and thus under heavy traffic with query restrictions (1 week at a time currently). We also archive data that is older than 6 months so that the database stays lean and mean. This serves the purpose well for general day to day but limits us in terms of doing larger queries for the sake of summary statistics and billing. What I would like to stand up is a second slurmdbd that has no query restrictions and has all my historical data, while also being kept abreast of the live information coming off the cluster. Straight replication (slurmdbd or mysql) won't suffice as when I do my monthly purges, it would then propagate that purge. Likewise reimporting the archived data into an external database would be missing the current 6 months. My question is is there a clever way to run a duplicate slurmdbd that gets all the cluster traffic so it stays up to date, while also not purging data? Also in such as way that I could say have the duplicate slurmdbd offsite and not drag down cluster performance (as really the cluster should lean on the active database that is more proximate)? Looking at the documentation I can see an obvious method for HA, and AccountingStorageExternalHost which looks like it might work but may drag down performance due to needing to wait for DB operations to complete there. I'm curious if other sites have solved this. Summary programs like XDMod are useful but having the raw data available to cross check and process in new ways is what we are really looking for. Thanks in advance. -Paul Edmon-

4 4

How Should I Size CPU and RAM for New Slurm Compute Nodes?
by wesleycolton238＠gmail.com 23 Sep '26

23 Sep '26

We’re planning to add a few new compute nodes to a small Slurm cluster, and I’m trying to work out a sensible CPU-to-memory balance before choosing the hardware. My current thought is to look at historical job data first, especially allocated CPUs, requested memory, MaxRSS, runtime, and how often jobs are waiting because of CPU versus memory availability. That seems more useful than simply buying the highest core-count CPUs available. For the new nodes, I’m mainly trying to decide: Whether fewer higher-clocked cores or more lower-clocked cores would give us better overall throughput How much RAM per physical core makes sense based on actual job usage Whether it is better to standardize all nodes or have a separate high-memory partition How much physical RAM should be left outside Slurm's RealMemory value for the OS and system processes Whether NUMA layout and memory bandwidth should influence the CPU choice as much as core count I’ve also been comparing different rack-server configurations here to get an idea of available Xeon, ECC memory, and storage combinations: https://ormsystems.com/servers-price.html For those managing production Slurm clusters, do you normally size new nodes from something like the 80th/90th percentile of historical memory-per-core usage, or do you use a different method? I’d be interested to hear what metrics you’ve found most useful before purchasing additional compute nodes.

3 2

How do I get the information available from `scontrol show job` in a format I can safely parse?
by Adam Novak 23 Sep '26

23 Sep '26

It was recently pointed out to me that `scontrol show job` will include jobs' comments, and those comments seem to be able to contain arbitrary text, including newlines and "=". The comments don't seem to be delimited or escaped in any way within the `scontrol show job` output. This means I can't actually parse the output of `scontrol show job`, because once a comment starts I can never tell that it has ended. Someone can put a whole valid-looking job record inside their job comment, or something that looks like the tail end of one job record and the beginning of another. Is there a way to adjust the output format or access that data, as a normal cluster user, through another interface? To illustrate the problem, I ran: sbatch --wrap "sleep 10" --comment="$(printf "\n\nJobID=12345\n\n")" And I found this in my `scontrol show job` output: JobId=38807436 JobName=wrap UserId=anovak1(30065) GroupId=prismuser(600) MCS_label=N/A Priority=4202 Nice=0 Account=standard QOS=normal JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:09 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2026-09-22T13:47:15 EligibleTime=2026-09-22T13:47:15 AccrueTime=2026-09-22T13:47:15 StartTime=2026-09-22T13:47:17 EndTime=2026-09-22T13:57:17 Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-09-22T13:47:17 Scheduler=Backfill Partition=short AllocNode:Sid=phoenix:1297984 ReqNodeList=(null) ExcNodeList=(null) NodeList=phoenix-06 BatchHost=phoenix-06 NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* TRES=cpu=1,mem=4000M,node=1,billing=1 Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=4000M MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=(null) WorkDir=/private/home/anovak1/workspace/cwltool Comment= JobID=12345 StdErr=/private/home/anovak1/workspace/cwltool/slurm-38807436.out StdIn=/dev/null StdOut=/private/home/anovak1/workspace/cwltool/slurm-38807436.out Power= -- Adam Novak (He/Him) Senior Software Engineer Computational Genomics Lab UC Santa Cruz Genomics Institute "Revealing life’s code." Personal Feedback: https://forms.gle/UXZhZc123knF65Dw5

6 7

Timed-out jobs were 6.6% of jobs but ~45% of job energy on two DOE clusters
by James Jardine 22 Sep '26

22 Sep '26

Hello, I've been working through job-scheduler records from two DOE clusters (Eagle and Kestrel) and one result seems worth putting in front of this list. Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of total job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing. The part I didn't expect is how predictable they are before they start. Four summary statistics (mean, standard deviation, range, mean absolute successive difference) over a user's previous 24 Elapsed/Timelimit ratios, plain logistic regression, using nothing at all about the job being predicted. Forward-chained by month, never trained on anything later than the month being scored. The prediction figures are Kestrel only: about 6.8M job windows over 22 monthly splits. At an 80th-percentile threshold, pooled across splits, 95.4% of timed-out jobs were flagged, at the cost of flagging 16.9% of jobs that would have finished fine. Per split that is 93.2% +/- 6.3 and 21.3% +/- 12.5. The false-flag spread is wide, and it is the number that decides whether this is usable at a given site, so I would look at that before the headline. The model is deliberately boring, and that is the point. What took the work was the measurement around it: forward-chaining by month so nothing is ever trained on the future, a label-shuffle control that has to land at chance before any number is printed, and a refusal path so it prints nothing at all when the data won't carry a conclusion. 868 lines total, so you can audit the lot in an afternoon. I packaged it so you can run it against your own sacct export: git clone https://github.com/JJardine919/lattice24-assess cd lattice24-assess && python3 -m venv .venv && .venv/bin/pip install -e . sacct --allocations --parsable2 --starttime=$(date -d '12 months ago' +%F) \ -o User,End,Timelimit,Elapsed,State,ConsumedEnergyRaw > export.psv .venv/bin/lattice24-assess export.psv --out ./report If your site already has numpy and scikit-learn available, skip the venv and run python3 -m lattice24_assess.cli export.psv --out ./report from the clone. It runs entirely on the login node. There are no network calls in it, and the source is short enough to confirm that yourself in a couple of minutes. User IDs are hashed with a per-run salt; job names, working directories and command lines are dropped at parse time and never read. It refuses rather than degrades: under 6 months of data, under 50k windows, under 200 timeout events, or if the label-shuffle control doesn't land near chance, it explains why and prints no headline number. Two things it deliberately will not tell you. It reports energy AT STAKE, not energy saved -- a flag avoids nothing until somebody resizes, fixes or declines the job, and that multiplier belongs to your site, not to me. And it covers TIMEOUT only; CANCELLED and FAILED are excluded, so your recoverable waste is probably larger than what it reports. Repo, Apache-2.0: https://github.com/JJardine919/lattice24-assess Method and results: https://doi.org/10.5281/zenodo.21911351 I'd genuinely like to know whether the pattern holds elsewhere. If your number comes out much lower, that is a useful result and I'd like to hear it. Jim Jardine Lattice24

2 3

Suspending jobs when preempted
by pie＠lem.com 13 Sep '26

13 Sep '26

I am new to Slurm, sorry if my question has been asked before, I could only find some older discussions about this. I am currently setting up a job scheduler in our company. Main bottleneck is licenses, so a license scheduler including preemption should be included. I have noticed this in the documentation: "Because licenses are not freed when jobs are suspended, jobs using licenses requested by higher priority jobs will only be prempted when PreemptMode is either REQUEUE or CANCEL and PreemptParameters=reclaim_licenses is set." Which means I cannot set the PreemptMode to SUSPEND, is that correct? Is that something that will ever be implemented? I have created tres licences and implemented a polling mechanism that keeps the slurm tres licenses in sync with the real licenses. Our application automatically releases licenses when jobs get a a TSTP signal. So, all I want is the Slurm scheduler to suspend low prio jobs when needed, send a TSTP signal and let the book-keeping be done by my polling script. Is this in any way possible?

2 3

reserving TRES(gpus) fails via nodes busy
by Äkäslompolo Simppa 04 Sep '26

04 Sep '26

Hi! I've been trying to reserve a couple of GPUs permanently for a specific user, but I keep getting "Requested nodes are busy" (slurm 26.05.3) Perhaps someone could give ideas what debug flags to enable or other ways to fish out more information why slurm says the nodes are busy? Here is an example case, but I've not figured out the systematics: (And flags=ignore_jobs does not make a difference) # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g,gres/gpu:h200:4 corecnt=48 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g corecnt=48 Reservation created: llmgw_permanent # Here is another case: # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:4 flags=ignore_jobs, nodes=gpu49 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:0 flags=ignore_jobs, nodes=gpu49 Reservation created: llmgw_permanent # Thank you! -- - Simppa - Mr. Simppa Äkäslompolo High performance computing specialist Doctor of Science (Tech.) Aalto Scientific Computing School of Science, Aalto University, Finland +358-50-5311327 https://scicomp.aalto.fi/

2 1

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available (CVE-2026-65107 et seq)
by Tim Wickberg 02 Sep '26

02 Sep '26

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available and include a number of security fixes (CVE-2026-65107 et seq). Slurm support customers were informed on August 19th and provided patches on request. Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases The following excerpt from the changelogs summarizes the issues and their associated CVE ids: * Fix possible slurmstepd crash on invalid step socket requests. CVE-2026-65168. * Fix sbcast shared objects skipping credential verification. CVE-2026-65107. * Fix possible slurmd crash on invalid sbcast filenames. CVE-2026-65107. * Fix a slurmstepd stack overflow when a job environment contains an oversized SPANK option variable. CVE-2026-65108. * Fix slurmstepd removing files outside the container spool directory when cleaning up an OCI container. CVE-2026-65109. * Fix slurmstepd leaving OCI container spool directories behind when ContainerPath contains a task id pattern. CVE-2026-65109. * Fix heap over-read when unpacking a malformed forward data RPC in slurmd. CVE-2026-65138. * Fix a slurmd crash when handling a malformed forward data RPC with a missing socket address. CVE-2026-65138. * Reject cluster names containing characters that are unsafe to use in an accounting database query. CVE-2026-65139. * Reject unsafe cluster names on the accounting usage, add and runaway job paths. CVE-2026-65139. * Reject unsafe cluster names when opening a connection to the slurmdbd. CVE-2026-65139. * Reject non-numeric id values in accounting database queries. CVE-2026-65139. * Fix a privilege escalation where an operator could alter Administrator accounts through the accounting database. CVE-2026-65140. * Fix node count of a job step using arbitrary distribution. CVE-2026-65165. * Reject a hostlist function in the node list of a job step using arbitrary distribution. CVE-2026-65165. * Reject a job step whose arbitrary node list disagrees with its node count. CVE-2026-65165. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md

1 0

Any update versions of SLURM soon?
by Grigory Shamov 02 Sep '26

02 Sep '26

HI All, I have heard that a minor SLURM update is coming. Just wondering if the public Githib repo of SchedMD/slurm still gets updated releases? -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

2 1

HPC Client GUI: desktop client for SSH and Slurm workflows — feedback welcome
by Mehmed Sinan KömeK 28 Aug '26

28 Aug '26

Hello everyone, I would like to introduce HPC Client GUI, a desktop application I have been developing to make everyday work with SSH-accessible Slurm clusters easier. The project grew out of my own research and CFD workflows on Türkiye’s TRUBA infrastructure. I wanted to reduce the need to switch between a terminal, an SFTP client, scheduler commands, and job-output files. HPC Client GUI brings these tasks together: - SSH connections and reusable connection profiles - Remote file browsing, SFTP uploads/downloads, and transfer management - Slurm job submission, queue monitoring, accounting information, and cancellation - Job output and error-file inspection - Integrated terminal and CLI workflows - Optional X11 forwarding - Downloadable cluster profiles, job templates, and lint rule packs, including ANSYS Fluent journal checks The application runs locally on Windows, macOS, and Linux. It uses SSH, SFTP, and standard Slurm commands such as sbatch, squeue, sacct, and scancel; it does not require a server-side agent, web portal, or slurmrestd deployment. Repository, screenshots, documentation, and downloads: https://github.com/mskomek/hpc-client-gui Although TRUBA was the original use case, the project is intended for other SSH + Slurm environments. Compatibility with individual clusters depends on their authentication methods, scheduler configuration, and site policies. This is an independent project, not an official TRUBA or SchedMD tool. I would appreciate feedback from both researchers and cluster administrators, especially on: - Compatibility with different Slurm configurations and output formats - SSH authentication, MFA, and jump-host requirements - Appropriate polling frequency and login-node load - Site policies or safeguards a desktop client should respect - Missing features or documentation that would prevent adoption If anyone is willing to try it on another cluster, the repository includes a read-only compatibility validation guide. Please do not share credentials, confidential hostnames, or private configuration in public reports. For transparency, the project is source-available under the PolyForm Noncommercial License 1.0.0; commercial use requires a separate license. The current macOS packages are unsigned and not notarized, as disclosed in the release notes. Thank you for your time. Suggestions, compatibility reports, and critical feedback are all welcome. Best regards, Mehmed Sinan Kömek

1 0

cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup
by Xaver Stiensmeier 24 Aug '26

24 Aug '26

Dear slurm-user list, currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed"). Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue: [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18 My *cgroup.conf*: CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100 From *slurm.conf*: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup *Slurm Version: *slurm 24.11.3 As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas! Best, Xaver

3 4

Slurm version 26.05.3 is now available
by Tim McMullan 13 Aug '26

13 Aug '26

We are pleased to announce the availability of Slurm version 26.05.3. This release fixes some mild to moderate severity issues and crashes, including srun potentially hanging during step complete due to a race and emitting a new warning when innodb_snapshot_isolation is set to ON. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Epilog error
by Ratnasamy, Fritz 13 Aug '26

13 Aug '26

Hello, Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n" if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Job stuck in pending state with AssocGrpMemLimit
by Rémy Dernat 10 Aug '26

10 Aug '26

Hi ! I have a very strange behavior that I am unable to explain. A user have a job stuck in pending state, although there are enough resources to run : ######################################### [dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools [dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit) 8936077 cpu-dedic sim11.13 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930985 cpu-dedic sim33.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930981 cpu-dedic sim32.14 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930980 cpu-dedic sim31.61 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930979 cpu-dedic sim23.21 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930976 cpu-dedic sim22.27 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930972 cpu-dedic sim21.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930971 cpu-dedic sim13.24 gammaPD 0:00 2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315 1420G omega 1294 6 8880518 5000M gamma1326 191 [dernatr@io-login-01 ~]$ scontrol show job 8949801_0 JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9 UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53 AccrueTime=2026-08-03T16:59:53 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=560G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/scratch/users/doej/run_access_d49_23.sh WorkDir=/scratch/users/doej AdminComment=159722b1-908e-47e2-82b1-08872ee64906 StdErr=/scratch/users/doej/slurm-8949801_0.err StdIn=/dev/null StdOut=/scratch/users/doej/slurm-8949801_0.out TresPerTask=cpu=1 [dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup Account Org Descr GrpTRES Share MaxWall Def QOS QOS ------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate [dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb DefMemPerCPU = 2048 MaxMemPerNode = UNLIMITED PropagateResourceLimitsExcept = MEMLOCK SelectTypeParameters = CR_CORE_MEMORY MemorySwappiness = (null) [dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501 dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119 dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353 dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971 dedicated-cpu@igroup 1 0.030303 232632330 0.129448 Account User RawShares NormShares RawUsage EffectvUsage FairShare -------------------- ---------- ---------- ----------- ----------- ------------- ---------- ######################################### As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB. I also look at backfilling and fair sharing, but I don't see anything that could block him. Any idea may be useful, Thanks, Best regards, R -- ISDM Lundi/Mardi/Jeudi/Vendredi ISEM Mercredi

5 4

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 10 Aug '26

10 Aug '26

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

4 5

sreport changes between Slurm 24 and 26?
by Kamil Wilczek 20 Jul '26

20 Jul '26

Dear All, I recently upgraded my cluster to Slurm 26.05.2 (a completely fresh installation, including the operating system), replacing a previous Slurm 24.11 installation. On the previous installation (Slurm 24.11), I used sreport from a Python script to retrieve TRES usage per QOS: slurm_sreport = subprocess.run( [ "sreport", "--tres=cpu,gres/gpu", "cluster", "AccountUtilizationByQOS", "Start=2025-01-01", "End=now", "Accounts=mim", "Format=QOS,TresName,Used", f"Users={os.environ['USER']}", "--noheader", "--parsable2" ], ... The script generated a nice report for each user who ran it. The "Users=" parameter was respected, and the command returned only QOSes associated with the calling user. With Slurm 26.05.2, however, Users= appears to be ignored, and the command returns the complete list of QOSes in the cluster instead. Is this a confirmed change in behaviour? Kind regards, -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Dummy Slurm jobs?
by Grigory Shamov 18 Jul '26

18 Jul '26

HI All, Is it possible, or have it been done, to create dummy Slurm jobs that would start, block the resource, execute prologue and epilogue, but never execute the batch script body? The idea is to start something in prologue, like a user/job/credential-specific instance of a proxy (say, nginx) controlled/scheduled by Slurm, but without giving users ability to run an arbitrary script in the job's script body. Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

3 2

Slurm versions 26.05.2 and 25.11.7 are now available
by Tim McMullan 14 Jul '26

14 Jul '26

We are pleased to announce the availability of Slurm versions 26.05.2 and 25.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Cluster ID changed in Slurm DB
by Burian, John 10 Jul '26

10 Jul '26

I was working on adding some partitions to our test cluster to test preemption. This is Slurm 24.11.1, we have a single slurmctld and slurmdbd in this cluster. I’m using partition based preemption, not QOS based preemption. I added a QOS to use with the preempting partition to enforce per-job resource limits, and then added the new partitions to slurm.conf. It took me many tries to get the slurm.conf correct, so many attempts to start slurmctld that failed due to conf file parsing errors. However, once I got the slurm.conf correct, slurmctld would not start, complaining that the cluster ID in the saved state information was wrong: [2026-07-02T14:55:38.573] fatal: CLUSTER ID MISMATCH. slurmctld has been started with "ClusterID=3013" from the state files in StateSaveLocation, but the DBD thinks it should be "3391". Running multiple clusters from a shared StateSaveLocation WILL CAUSE CORRUPTION. Remove /export/slurm/state/clustername to override this safety check if this is intentional. The clustername file in the save state location shows the cluster ID 3013. In the Slurm DB, in clusters_table, I see the row for my test cluster shows the (wrong) ID that appears in the logs, 3391, and the mod_time field has a timestamp in the same time frame as when I was doing the work on adding the QOS and partition configuration. Any thoughts on what could have caused the Slurm DB to be updated with a different cluster ID than what is saved in state? I’d like to avoid something similar happening when I go to deploy the new partitions in production. John

1 0

Expected behavior for preempt/partition_prio
by Massimo Sgaravatto 10 Jul '26

10 Jul '26

Dear all I have these settings in slurm.conf: . ... PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=gpus-opp Nodes=btc-ter-gpu-[03-05],btc-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=15 PartitionName=mpicms Nodes=btc-ter-03,btc-ter-gpu-05 AllowAccounts=cms,operators State=UP PriorityTier=20 .... My understanding was that the expected behavior is that a job submitted to the gpus-opp partition is always preempted (requeued) if the resources allocated by this job are requested by a job submitted to the mpicms partition (since 20 > 15) Did I get it right? I am asking because I see that sometimes this doesn't happen. I am still trying to debug the issue, but it looks like also the QoS is taken into account (and this doesn't make sense to me unless I am missing something) Thanks, Massimo

3 3

Slinky 1.2.0 released
by Marlow Warnicke 02 Jul '26

02 Jul '26

We are pleased to announce the availability of Slinky version 1.2.0! Slinky is SchedMD (now part of NVIDIA)'s set of components to integrate Slurm in Kubernetes environments. Slinky consists of two main projects, slurm-operator and slurm-bridge. Our landing page is here: https://www.slinky.ai The slurm-operator handles cases where users wish to run Slurm jobs within the Kubernetes cluster. Release v1.2.0: https://github.com/SlinkyProject/slurm-operator/releases/tag/v1.2.0 New features include a ScheduledUpdate strategy for NodeSets to control when updates are applied, support for Google GKE A3 Mega instance types, and configurable pod security contexts for operator and webhook deployments. The full changelog may be found here: https://github.com/SlinkyProject/slurm-operator/blob/main/CHANGELOG/CHANGEL… The slurm-bridge brings Kubernetes and Slurm jobs together under one scheduler. Both types of workloads share the same cluster resources. Release v1.2.0: https://github.com/SlinkyProject/slurm-bridge/releases/tag/v1.2.0 New features include support for LeaderWorkerSet workloads introduced in Kubernetes 1.37, priority annotations for workloads to control Slurm job scheduling order, and dynamic topology reconciliation for external and hybrid Slurm nodes. The full changelog may be found here: https://github.com/SlinkyProject/slurm-bridge/blob/main/CHANGELOG/CHANGELOG… #kubernetes #slurm #dra #ai #training #inference #hpc #nvidia #gpu #cloudnative #slinky

1 0

Avoid killing GPU jobs during preemption
by Ratnasamy, Fritz 29 Jun '26

29 Jun '26

Hello, We have a recurring demand where some users would like to use idle GPU resources on periods where the usage is low. Their normal GPU/qos limits prevent this, which is expected as part of the fair share scheduler principles. A pattern we're considering is the preempt option on slurm to have these jobs being pre-empted when another user is submitting his jobs within his quota. The jobs that were submitted earlier by a faculty member taking advantage of available resources on top of his regular qos, will be preempted however, there is no suspend option for gpu jobs. There is only requeue, which kills the jobs to make the gpu available. I believe you can use suspend for cpu/mem without killing the job. I know that would require the users to set up checkpoints and re-start their jobs. The goal is to let heavy users fill otherwise idle nodes over low usage time without impacting smaller users when activity picks up. Has anyone implemented something similar? I am in general against preemption because that might lead to conflict between users. Thanks, *Fritz Ratnasamy*Data Scientist Information Technology

2 1

ISC 2026 is here! Come see us in Hamburg.
by Victoria Hobson 22 Jun '26

22 Jun '26

The wait is over! ISC 2026 is officially underway in Hamburg, Germany, and we'd love to see you! As part of the newly expanded NVIDIA family, the Slurm team is on the floor and ready to connect. Swing by the Slurm booth (Z10) for sessions on GPU workload management, Slinky, and the latest releases and don't miss the NVIDIA booth (E30) for even more great content all week long. Here's what's on tap at booth Z10: Tuesday, June 23 12:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 17:00 – What's New: Latest Slurm and Slinky Release 19:00 – Slinky: Slurm for Kubernetes 101 Wednesday, June 24 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement 14:00 – Slinky: Slurm for Kubernetes 101 15:00 – What's New: Latest Slurm and Slinky Release 17:00 – Redefining Resiliency for HPC Research and AI Factories: Slurm and NVIDIA Mission Control Thursday, June 25 10:00 – Slinky: Slurm for Kubernetes 101 11:00 – Mastering GPU Topology: Optimize Multi-Node Workload Placement Come say hello — we'll see you on the floor! Victoria Hobson | NVIDIA Campaign Manager, CUDA and Slurm

1 0

slurmctld keeps segfaulting, possibly during or just after backfill
by Marcus Lauer 21 Jun '26

21 Jun '26

We are running into a problem where slurmctld is segfaulting a few times a day. We had this problem with SLURM 23.11.8 and now with 23.11.10 as well, though the problem only appears on one of the several SLURM clusters we have, and all of them use one of those versions of SLURM. I was wondering if anyone has encountered a similar issue and has any thoughts on how to prevent this. Obviously we use "SchedulerType=sched/backfill" but strangely when I switched to sched/builtin for a while there were still slurmctld segfaults. We also set "SchedulerParameters=enable_user_top,bf_max_job_test=2000". I have tried turning those off but it did not help. I have also tried tweaking several other settings to no avail. Most of the cluster runs Rocky Linux 8.10 (including the slurmctld system) though we still have some Scientific Linux 7.9 compute nodes (we compile SLURM separately for those). Here is the crash-time error from journalctl: Oct 02 06:31:20 our.host.name kernel: sched_agent[2048355]: segfault at 8 ip 00007fec755d7ea8 sp 00007fec6bffe7e8 error 4 in libslurmfull.so[7fec7555a000+1f4000] Oct 02 06:31:20 our.host.name kernel: Code: 48 39 c1 7e 19 48 c1 f8 06 ba 01 00 00 00 48 d3 e2 48 f7 da 48 0b 54 c6 10 48 21 54 c7 10 c3 b8 00 00 00 00 eb da 48 8b 4f 08 <48> 39 4e 08 48 0f 4e 4e 08 49 89 c9 48 83 f9 3f 76 4e ba 40 00 00 Oct 02 06:31:20 our.host.name systemd[1]: Started Process Core Dump (PID 2169426/UID 0). Oct 02 06:31:20 our.host.name systemd-coredump[2169427]: Process 2048344 ( slurmctld) of user 991 dumped core. This is followed by a list of each of the dozen or so related threads. The one which is dumping core is first and looks like this: Stack trace of thread 2048355: #0 0x00007fec755d7ea8 bit_and_not (libslurmfull.so) #1 0x000000000044531f _job_alloc (slurmctld) #2 0x000000000044576b _job_alloc_whole_node_internal (slurmctld) #3 0x0000000000446e6d gres_ctld_job_alloc_whole_node (slurmctld) #4 0x00007fec722e29b8 job_res_add_job (select_cons_tres.so) #5 0x00007fec722f7c32 select_p_select_nodeinfo_set (select_cons_tres.so) #6 0x00007fec756e7dc7 select_g_select_nodeinfo_set (libslurmfull.so) #7 0x0000000000496eb3 select_nodes (slurmctld) #8 0x0000000000480826 _schedule (slurmctld) #9 0x00007fec753421ca start_thread (libpthread.so.0) #10 0x00007fec745f78d3 __clone (libc.so.6) I have run slurmctld with "debug5" level logging and it appears that the error occurs right after backfill considers a large number of jobs. Slurmctld could be failing at the end of backfill or when doing something which happens just after backfill runs. Usually this is the last message before the crash: [2024-09-25T18:39:42.076] slurmscriptd: debug: _slurmscriptd_mainloop: finished If anyone has any thoughts or advice on this that would be appreciated. Thank you. -- Marcus Lauer Systems Administrator CETS Group, Research Support

4 4

controller node in Ubuntu 22.04.5 LTS does not work with compute node in Debian GNU/Linux 12
by allspace＠qq.com 17 Jun '26

17 Jun '26

download slurm-26.05.1.tar.bz2 and build it on Ubuntu 22.04.5 LTS. Use it as controller node. download slurm-26.05.1.tar.bz2 and build it on Debian GNU/Linux 12. Use it as compute node(node-3). Run command "srun -w node-3 hostname" on controller node, got the following error message: srun: error: Task launch for StepId=10.0 failed on node node-3: Header lengths are longer than data received srun: error: Application launch failed: Header lengths are longer than data received srun: Job step aborted Here is the logs on node-3: [2026-06-16T12:45:20.645] error: _verify_signature: failed decode [2026-06-16T12:45:20.645] error: Malformed RPC of type REQUEST_LAUNCH_TASKS(6001) received [2026-06-16T12:45:20.653] error: slurm_unpack_msg_and_forward: [192.168.245.1:59308] failed: Header lengths are longer than data received [2026-06-16T12:45:20.663] error: wrap_on_data: [192.168.245.133:6818(fd:13)] on_data returned rc: Header lengths are longer than data received How can I fix this problem? Thanks

5 7