August 2026 - slurm-users - lists.schedmd.com

Slurm version 26.05.3 is now available
by Tim McMullan 13 Aug '26

13 Aug '26

We are pleased to announce the availability of Slurm version 26.05.3. This release fixes some mild to moderate severity issues and crashes, including srun potentially hanging during step complete due to a race and emitting a new warning when innodb_snapshot_isolation is set to ON. The full list of changes are available in the CHANGELOG files for each version: https://github.com/SchedMD/slurm/blob/slurm-26.05/CHANGELOG/slurm-26.05.md Slurm can be downloaded from: https://github.com/SchedMD/slurm/releases -- Tim McMullan Senior System Software Engineer NVIDIA

1 0

Epilog error
by Ratnasamy, Fritz 13 Aug '26

13 Aug '26

Hello, Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n" if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? Best, *Fritz Ratnasamy*Data Scientist Information Technology

3 2

Job stuck in pending state with AssocGrpMemLimit
by Rémy Dernat 10 Aug '26

10 Aug '26

Hi ! I have a very strange behavior that I am unable to explain. A user have a job stuck in pending state, although there are enough resources to run : ######################################### [dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools [dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit) 8936077 cpu-dedic sim11.13 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930985 cpu-dedic sim33.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930981 cpu-dedic sim32.14 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930980 cpu-dedic sim31.61 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930979 cpu-dedic sim23.21 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930976 cpu-dedic sim22.27 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930972 cpu-dedic sim21.23 gammaPD 0:00 2 (AssocGrpCpuLimit) 8930971 cpu-dedic sim13.24 gammaPD 0:00 2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315 1420G omega 1294 6 8880518 5000M gamma1326 191 [dernatr@io-login-01 ~]$ scontrol show job 8949801_0 JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9 UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53 AccrueTime=2026-08-03T16:59:53 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=560G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/scratch/users/doej/run_access_d49_23.sh WorkDir=/scratch/users/doej AdminComment=159722b1-908e-47e2-82b1-08872ee64906 StdErr=/scratch/users/doej/slurm-8949801_0.err StdIn=/dev/null StdOut=/scratch/users/doej/slurm-8949801_0.out TresPerTask=cpu=1 [dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup Account Org Descr GrpTRES Share MaxWall Def QOS QOS ------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate [dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb DefMemPerCPU = 2048 MaxMemPerNode = UNLIMITED PropagateResourceLimitsExcept = MEMLOCK SelectTypeParameters = CR_CORE_MEMORY MemorySwappiness = (null) [dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501 dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119 dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353 dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971 dedicated-cpu@igroup 1 0.030303 232632330 0.129448 Account User RawShares NormShares RawUsage EffectvUsage FairShare -------------------- ---------- ---------- ----------- ----------- ------------- ---------- ######################################### As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB. I also look at backfilling and fair sharing, but I don't see anything that could block him. Any idea may be useful, Thanks, Best regards, R -- ISDM Lundi/Mardi/Jeudi/Vendredi ISEM Mercredi

5 4

Nodes Become Invalid Due to Less Total RAM Than Expected
by Xaver Stiensmeier 10 Aug '26

10 Aug '26

Dear slurm-user list, in the past we had a bigger buffer between RealMemory <https://slurm.schedmd.com/slurm.conf.html#OPT_RealMemory> and the instance memory. We then discovered that the right way is to activating the *memory option* (SelectTypeParameters=CR_Core_Memory) and setting MemSpecLimit <https://slurm.schedmd.com/slurm.conf.html#OPT_MemSpecLimit> to secure RAM for system processes. However, now we run into the problem that due to *on demand scheduling*, we have to setup the slurm.conf in advance by using the RAM values from our flavors as reported by our cloud provider (OpenStack). These RAM values are higher than the RAM values the machines actually have later on: ram_in_mib by openstack total_ram_in_mib by top/slurm 2048 1968 16384 15991 32768 32093 65536 64297 122880 120749 245760 241608 491520 483528 Given that we have to define the slurm.conf in advance, we kinda have to predict how much total ram the instances have once created. Of course I used linear regression to approximate the total ram and then lowered it a bit to have some cushion, but this feels unsafe given that future flavors could differ from that. From the kernel documentation <https://www.kernel.org/doc/Documentation/filesystems/proc.txt> I know that MemTotal is MemTotal: Total usable ram (i.e. physical ram minus a few reserved bits and the kernel binary code) but given that the concrete reserved bits are quite complex <https://witekio.com/blog/cat-proc-meminfo-memtotal/>, I am wondering whether I am doing something wrong as this issue doesn't feel niche enough to be that complicated. --- Anyway, setting the RAM value in the slurm.conf above total ram by predicting too much, leads to errors and nodes being marked as invalid: [2025-08-11T08:19:04.736] debug: Node NODE_NAME has low real_memory size (241607 / 245760) < 100.00% [2025-08-11T08:19:04.736] error: _slurm_rpc_node_registration node=NODE_NAME: Invalid argument or |[2025-07-03T12:57:18.486] error: Setting node NODE_NAME state to INVAL with reason:Low RealMemory (reported:64295 < 100.00% of configured:68719)| |Any hint on how to solve this is much appreciated! | Best regards, Xaver

4 5