- slurm-users - lists.schedmd.com

reserving TRES(gpus) fails via nodes busy
by Äkäslompolo Simppa 03 Sep '26

03 Sep '26

Hi! I've been trying to reserve a couple of GPUs permanently for a specific user, but I keep getting "Requested nodes are busy" (slurm 26.05.3) Perhaps someone could give ideas what debug flags to enable or other ways to fish out more information why slurm says the nodes are busy? Here is an example case, but I've not figured out the systematics: (And flags=ignore_jobs does not make a difference) # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g,gres/gpu:h200:4 corecnt=48 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw nodes=gpu49 start=2026-09-02T10:00 duration=infinite TRES=mem=800g corecnt=48 Reservation created: llmgw_permanent # Here is another case: # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:4 flags=ignore_jobs, nodes=gpu49 Error creating the reservation: Requested nodes are busy # scontrol create reservation reservationname=llmgw_permanent user=llmgw partition=gpu-h200-141g-short start=now duration=infinite TRES=mem=800g,gres/gpu:h200:0 flags=ignore_jobs, nodes=gpu49 Reservation created: llmgw_permanent # Thank you! -- - Simppa - Mr. Simppa Äkäslompolo High performance computing specialist Doctor of Science (Tech.) Aalto Scientific Computing School of Science, Aalto University, Finland +358-50-5311327 https://scicomp.aalto.fi/

1 0

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available (CVE-2026-65107 et seq)
by Tim Wickberg 02 Sep '26

02 Sep '26

Slurm versions 26.05.4, 25.11.8, and 25.05.9 are now available and include a number of security fixes (CVE-2026-65107 et seq). Slurm support customers were informed on August 19th and provided patches on request. Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases The following excerpt from the changelogs summarizes the issues and their associated CVE ids: * Fix possible slurmstepd crash on invalid step socket requests. CVE-2026-65168. * Fix sbcast shared objects skipping credential verification. CVE-2026-65107. * Fix possible slurmd crash on invalid sbcast filenames. CVE-2026-65107. * Fix a slurmstepd stack overflow when a job environment contains an oversized SPANK option variable. CVE-2026-65108. * Fix slurmstepd removing files outside the container spool directory when cleaning up an OCI container. CVE-2026-65109. * Fix slurmstepd leaving OCI container spool directories behind when ContainerPath contains a task id pattern. CVE-2026-65109. * Fix heap over-read when unpacking a malformed forward data RPC in slurmd. CVE-2026-65138. * Fix a slurmd crash when handling a malformed forward data RPC with a missing socket address. CVE-2026-65138. * Reject cluster names containing characters that are unsafe to use in an accounting database query. CVE-2026-65139. * Reject unsafe cluster names on the accounting usage, add and runaway job paths. CVE-2026-65139. * Reject unsafe cluster names when opening a connection to the slurmdbd. CVE-2026-65139. * Reject non-numeric id values in accounting database queries. CVE-2026-65139. * Fix a privilege escalation where an operator could alter Administrator accounts through the accounting database. CVE-2026-65140. * Fix node count of a job step using arbitrary distribution. CVE-2026-65165. * Reject a hostlist function in the node list of a job step using arbitrary distribution. CVE-2026-65165. * Reject a job step whose arbitrary node list disagrees with its node count. CVE-2026-65165. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md https://github.com/SchedMD/slurm/blob/slurm-25.05/CHANGELOG/slurm-25.05.md

1 0

Any update versions of SLURM soon?
by Grigory Shamov 02 Sep '26

02 Sep '26

HI All, I have heard that a minor SLURM update is coming. Just wondering if the public Githib repo of SchedMD/slurm still gets updated releases? -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

2 1

HPC Client GUI: desktop client for SSH and Slurm workflows — feedback welcome
by Mehmed Sinan KömeK 28 Aug '26

28 Aug '26

Hello everyone, I would like to introduce HPC Client GUI, a desktop application I have been developing to make everyday work with SSH-accessible Slurm clusters easier. The project grew out of my own research and CFD workflows on Türkiye’s TRUBA infrastructure. I wanted to reduce the need to switch between a terminal, an SFTP client, scheduler commands, and job-output files. HPC Client GUI brings these tasks together: - SSH connections and reusable connection profiles - Remote file browsing, SFTP uploads/downloads, and transfer management - Slurm job submission, queue monitoring, accounting information, and cancellation - Job output and error-file inspection - Integrated terminal and CLI workflows - Optional X11 forwarding - Downloadable cluster profiles, job templates, and lint rule packs, including ANSYS Fluent journal checks The application runs locally on Windows, macOS, and Linux. It uses SSH, SFTP, and standard Slurm commands such as sbatch, squeue, sacct, and scancel; it does not require a server-side agent, web portal, or slurmrestd deployment. Repository, screenshots, documentation, and downloads: https://github.com/mskomek/hpc-client-gui Although TRUBA was the original use case, the project is intended for other SSH + Slurm environments. Compatibility with individual clusters depends on their authentication methods, scheduler configuration, and site policies. This is an independent project, not an official TRUBA or SchedMD tool. I would appreciate feedback from both researchers and cluster administrators, especially on: - Compatibility with different Slurm configurations and output formats - SSH authentication, MFA, and jump-host requirements - Appropriate polling frequency and login-node load - Site policies or safeguards a desktop client should respect - Missing features or documentation that would prevent adoption If anyone is willing to try it on another cluster, the repository includes a read-only compatibility validation guide. Please do not share credentials, confidential hostnames, or private configuration in public reports. For transparency, the project is source-available under the PolyForm Noncommercial License 1.0.0; commercial use requires a separate license. The current macOS packages are unsigned and not notarized, as disclosed in the release notes. Thank you for your time. Suggestions, compatibility reports, and critical feedback are all welcome. Best regards, Mehmed Sinan Kömek

1 0

cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup
by Xaver Stiensmeier 24 Aug '26

24 Aug '26

Dear slurm-user list, currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed"). Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue: [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18 My *cgroup.conf*: CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100 From *slurm.conf*: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup *Slurm Version: *slurm 24.11.3 As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas! Best, Xaver

3 4

Slurm version 26.05.3 is now available
by Tim McMullan 13 Aug '26

13 Aug '26

We are pleased to announce the availability of Slurm version 26.05.3. This release fixes some mild to moderate severity issues and crashes, including srun potentially hanging during step complete due to a race and emitting a new warning when innodb_snapshot_isolation is set to ON. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Epilog error
by Ratnasamy, Fritz 13 Aug '26

13 Aug '26

Hello, Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n" if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Job stuck in pending state with AssocGrpMemLimit
by Rémy Dernat 10 Aug '26

10 Aug '26

Hi ! I have a very strange behavior that I am unable to explain. A user have a job stuck in pending state, although there are enough resources to run : ######################################### [dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools [dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit) 8936077 cpu-dedic sim11.13 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930985 cpu-dedic sim33.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930981 cpu-dedic sim32.14 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930980 cpu-dedic sim31.61 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930979 cpu-dedic sim23.21 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930976 cpu-dedic sim22.27 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930972 cpu-dedic sim21.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930971 cpu-dedic sim13.24 gammaPD 0:00 2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315 1420G omega 1294 6 8880518 5000M gamma1326 191 [dernatr@io-login-01 ~]$ scontrol show job 8949801_0 JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9 UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53 AccrueTime=2026-08-03T16:59:53 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=560G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/scratch/users/doej/run_access_d49_23.sh WorkDir=/scratch/users/doej AdminComment=159722b1-908e-47e2-82b1-08872ee64906 StdErr=/scratch/users/doej/slurm-8949801_0.err StdIn=/dev/null StdOut=/scratch/users/doej/slurm-8949801_0.out TresPerTask=cpu=1 [dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup Account Org Descr GrpTRES Share MaxWall Def QOS QOS ------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate [dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb DefMemPerCPU = 2048 MaxMemPerNode = UNLIMITED PropagateResourceLimitsExcept = MEMLOCK SelectTypeParameters = CR_CORE_MEMORY MemorySwappiness = (null) [dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501 dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119 dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353 dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971 dedicated-cpu@igroup 1 0.030303 232632330 0.129448 Account User RawShares NormShares RawUsage EffectvUsage FairShare -------------------- ---------- ---------- ----------- ----------- ------------- ---------- ######################################### As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB. I also look at backfilling and fair sharing, but I don't see anything that could block him. Any idea may be useful, Thanks, Best regards, R -- ISDM Lundi/Mardi/Jeudi/Vendredi ISEM Mercredi

5 4

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 10 Aug '26

10 Aug '26

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

4 5

sreport changes between Slurm 24 and 26?
by Kamil Wilczek 20 Jul '26

20 Jul '26

Dear All, I recently upgraded my cluster to Slurm 26.05.2 (a completely fresh installation, including the operating system), replacing a previous Slurm 24.11 installation. On the previous installation (Slurm 24.11), I used sreport from a Python script to retrieve TRES usage per QOS: slurm_sreport = subprocess.run( [ "sreport", "--tres=cpu,gres/gpu", "cluster", "AccountUtilizationByQOS", "Start=2025-01-01", "End=now", "Accounts=mim", "Format=QOS,TresName,Used", f"Users={os.environ['USER']}", "--noheader", "--parsable2" ], ... The script generated a nice report for each user who ran it. The "Users=" parameter was respected, and the command returned only QOSes associated with the calling user. With Slurm 26.05.2, however, Users= appears to be ignored, and the command returns the complete list of QOSes in the cluster instead. Is this a confirmed change in behaviour? Kind regards, -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0