- slurm-users - lists.schedmd.com

cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup
by Xaver Stiensmeier 24 Aug '26

24 Aug '26

Dear slurm-user list, currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed"). Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue: [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18 My *cgroup.conf*: CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100 From *slurm.conf*: ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup *Slurm Version: *slurm 24.11.3 As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas! Best, Xaver

3 4

Slurm version 26.05.3 is now available
by Tim McMullan 13 Aug '26

13 Aug '26

We are pleased to announce the availability of Slurm version 26.05.3. This release fixes some mild to moderate severity issues and crashes, including srun potentially hanging during step complete due to a race and emitting a new warning when innodb_snapshot_isolation is set to ON. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Epilog error
by Ratnasamy, Fritz 13 Aug '26

13 Aug '26

Hello, Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n" if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Job stuck in pending state with AssocGrpMemLimit
by Rémy Dernat 10 Aug '26

10 Aug '26

Hi ! I have a very strange behavior that I am unable to explain. A user have a job stuck in pending state, although there are enough resources to run : ######################################### [dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools [dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit) 8936077 cpu-dedic sim11.13 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930985 cpu-dedic sim33.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930981 cpu-dedic sim32.14 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930980 cpu-dedic sim31.61 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930979 cpu-dedic sim23.21 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930976 cpu-dedic sim22.27 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930972 cpu-dedic sim21.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930971 cpu-dedic sim13.24 gammaPD 0:00 2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315 1420G omega 1294 6 8880518 5000M gamma1326 191 [dernatr@io-login-01 ~]$ scontrol show job 8949801_0 JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9 UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53 AccrueTime=2026-08-03T16:59:53 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=560G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/scratch/users/doej/run_access_d49_23.sh WorkDir=/scratch/users/doej AdminComment=159722b1-908e-47e2-82b1-08872ee64906 StdErr=/scratch/users/doej/slurm-8949801_0.err StdIn=/dev/null StdOut=/scratch/users/doej/slurm-8949801_0.out TresPerTask=cpu=1 [dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup Account Org Descr GrpTRES Share MaxWall Def QOS QOS ------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate [dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb DefMemPerCPU = 2048 MaxMemPerNode = UNLIMITED PropagateResourceLimitsExcept = MEMLOCK SelectTypeParameters = CR_CORE_MEMORY MemorySwappiness = (null) [dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501 dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119 dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353 dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971 dedicated-cpu@igroup 1 0.030303 232632330 0.129448 Account User RawShares NormShares RawUsage EffectvUsage FairShare -------------------- ---------- ---------- ----------- ----------- ------------- ---------- ######################################### As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB. I also look at backfilling and fair sharing, but I don't see anything that could block him. Any idea may be useful, Thanks, Best regards, R -- ISDM Lundi/Mardi/Jeudi/Vendredi ISEM Mercredi

5 4

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 10 Aug '26

10 Aug '26

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

4 5

sreport changes between Slurm 24 and 26?
by Kamil Wilczek 20 Jul '26

20 Jul '26

Dear All, I recently upgraded my cluster to Slurm 26.05.2 (a completely fresh installation, including the operating system), replacing a previous Slurm 24.11 installation. On the previous installation (Slurm 24.11), I used sreport from a Python script to retrieve TRES usage per QOS: slurm_sreport = subprocess.run( [ "sreport", "--tres=cpu,gres/gpu", "cluster", "AccountUtilizationByQOS", "Start=2025-01-01", "End=now", "Accounts=mim", "Format=QOS,TresName,Used", f"Users={os.environ['USER']}", "--noheader", "--parsable2" ], ... The script generated a nice report for each user who ran it. The "Users=" parameter was respected, and the command returned only QOSes associated with the calling user. With Slurm 26.05.2, however, Users= appears to be ignored, and the command returns the complete list of QOSes in the cluster instead. Is this a confirmed change in behaviour? Kind regards, -- Kamil Wilczek [https://keys.openpgp.org/] [D415917E84B8DA5A60E853B6E676ED061316B69B]

1 0

Dummy Slurm jobs?
by Grigory Shamov 18 Jul '26

18 Jul '26

HI All, Is it possible, or have it been done, to create dummy Slurm jobs that would start, block the resource, execute prologue and epilogue, but never execute the batch script body? The idea is to start something in prologue, like a user/job/credential-specific instance of a proxy (say, nginx) controlled/scheduled by Slurm, but without giving users ability to run an arbitrary script in the job's script body. Thanks! -- Grigory Shamov Site Lead / HPC Specialist University of Manitoba and DRI Alliance Canada

3 2

Slurm versions 26.05.2 and 25.11.7 are now available
by Tim McMullan 14 Jul '26

14 Jul '26

We are pleased to announce the availability of Slurm versions 26.05.2 and 25.11.7. These releases fix some mild to moderate severity issues and potential crashes. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md https://github.com/SchedMD/slurm/blob/slurm-25.11/CHANGELOG/slurm-25.11.md Slurm can be downloaded from: https://www.schedmd.com/download-slurm/ -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Cluster ID changed in Slurm DB
by Burian, John 10 Jul '26

10 Jul '26

I was working on adding some partitions to our test cluster to test preemption. This is Slurm 24.11.1, we have a single slurmctld and slurmdbd in this cluster. I’m using partition based preemption, not QOS based preemption. I added a QOS to use with the preempting partition to enforce per-job resource limits, and then added the new partitions to slurm.conf. It took me many tries to get the slurm.conf correct, so many attempts to start slurmctld that failed due to conf file parsing errors. However, once I got the slurm.conf correct, slurmctld would not start, complaining that the cluster ID in the saved state information was wrong: [2026-07-02T14:55:38.573] fatal: CLUSTER ID MISMATCH. slurmctld has been started with "ClusterID=3013" from the state files in StateSaveLocation, but the DBD thinks it should be "3391". Running multiple clusters from a shared StateSaveLocation WILL CAUSE CORRUPTION. Remove /export/slurm/state/clustername to override this safety check if this is intentional. The clustername file in the save state location shows the cluster ID 3013. In the Slurm DB, in clusters_table, I see the row for my test cluster shows the (wrong) ID that appears in the logs, 3391, and the mod_time field has a timestamp in the same time frame as when I was doing the work on adding the QOS and partition configuration. Any thoughts on what could have caused the Slurm DB to be updated with a different cluster ID than what is saved in state? I’d like to avoid something similar happening when I go to deploy the new partitions in production. John

1 0

Expected behavior for preempt/partition_prio
by Massimo Sgaravatto 10 Jul '26

10 Jul '26

Dear all I have these settings in slurm.conf: . ... PreemptType=preempt/partition_prio PreemptMode=REQUEUE PartitionName=gpus-opp Nodes=btc-ter-gpu-[03-05],btc-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=15 PartitionName=mpicms Nodes=btc-ter-03,btc-ter-gpu-05 AllowAccounts=cms,operators State=UP PriorityTier=20 .... My understanding was that the expected behavior is that a job submitted to the gpus-opp partition is always preempted (requeued) if the resources allocated by this job are requested by a job submitted to the mpicms partition (since 20 > 15) Did I get it right? I am asking because I see that sometimes this doesn't happen. I am still trying to debug the issue, but it looks like also the QoS is taken into account (and this doesn't make sense to me unless I am missing something) Thanks, Massimo

3 3