I don't suppose dedicated-cpu@igroup has any children? I can only imagine Slurm has a different idea of what the totals are than squeue_account_count_running_rss does. Or maybe you can tell us that over the past week you found some other (weirder?) problem.
Od: Rémy Dernat via slurm-users <slurm-users@lists.schedmd.com>
Poslano: torek, 4. avgust 2026 03:28
Za: slurm-users@lists.schedmd.com
Zadeva: [slurm-users] Job stuck in pending state with AssocGrpMemLimit
Hi !
I have a very strange behavior that I am unable to explain.
A user have a job stuck in pending state, although there are enough resources to run :
#########################################
[dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools
[dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup
Number of CPUs used by dedicated-cpu@igroup: 230
Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit)
8936077 cpu-dedic sim11.13 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930985 cpu-dedic sim33.23 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930981 cpu-dedic sim32.14 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930980 cpu-dedic sim31.61 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930979 cpu-dedic sim23.21 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930976 cpu-dedic sim22.27 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930972 cpu-dedic sim21.23 gamma PD 0:00 2 (AssocGrpCpuLimit)
8930971 cpu-dedic sim13.24 gamma PD 0:00 2 (AssocGrpCpuLimit)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C"
JOBID MIN_MEMORY USER UID CPUS
8946435 64G orbital 1064 32
8929322 480G doej 2746 1
8925315 1420G omega 1294 6
8880518 5000M gamma 1326 191
[dernatr@io-login-01 ~]$ scontrol show job 8949801_0
JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9
UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A
Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated
JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null)
Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53
AccrueTime=2026-08-03T16:59:53
StartTime=Unknown EndTime=Unknown Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main
Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082
ReqNodeList=(null) ExcNodeList=(null)
NodeList=
NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=1,mem=560G,node=1,billing=1
AllocTRES=(null)
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/scratch/users/doej/run_access_d49_23.sh
WorkDir=/scratch/users/doej
AdminComment=159722b1-908e-47e2-82b1-08872ee64906
StdErr=/scratch/users/doej/slurm-8949801_0.err
StdIn=/dev/null
StdOut=/scratch/users/doej/slurm-8949801_0.out
TresPerTask=cpu=1
[dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup
Account Org Descr GrpTRES Share MaxWall Def QOS QOS
------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- --------------------
dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate
[dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem
AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb
DefMemPerCPU = 2048
MaxMemPerNode = UNLIMITED
PropagateResourceLimitsExcept = MEMLOCK
SelectTypeParameters = CR_CORE_MEMORY
MemorySwappiness = (null)
[dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn
dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501
dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119
dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353
dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971
dedicated-cpu@igroup 1 0.030303 232632330 0.129448
Account User RawShares NormShares RawUsage EffectvUsage FairShare
-------------------- ---------- ---------- ----------- ----------- ------------- ----------
#########################################
As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB.
I also look at backfilling and fair sharing, but I don't see anything that could block him.
Any idea may be useful,
Thanks,
Best regards,
R
--
ISDM Lundi/Mardi/Jeudi/Vendredi
ISEM Mercredi