Hi Rémy, I might be missing something obvious, but I was curious about the difference between what squeue_account_count_running_rss reports and what Slurm is enforcing. The CPU counts line up correctly (32 + 1 + 6 + 191 = 230), so the script looks to be finding the right jobs. My understanding is that squeue's %m shows the per-CPU value for --mem-per-cpu jobs, so a straight sum of the column would under count things. So is 8880518 possibly a --mem-per-cpu job? 5000M against 191 CPUs made me wonder, and it's the only one of the four it could be: when requesting per-CPU, 8946435 would charge 2048G (64G x 32) and 8925315 would charge 8520G (1420G x 6), both well over the 3360G limit. If that's what's happening, usage would be around 2896G against the 3360G GrpTRES, leaving less than the 560G the pending job wants — and AssocGrpMemLimit would just be correct. Rather than trusting my math, though, scontrol on that job should answer it for us. It'll show either MinMemoryCPU or MinMemoryNode, and AllocTRES will give us the real total: scontrol show job 8880518 We can also check the association manager's own view of current usage against the limit: scontrol show assoc_mgr account=dedicated-cpu@igroup flags=assoc By the way, multi-node jobs would have a similar issue: %m is per node, so a two-node job charges twice what the column shows. Still on the child accounts question, maybe it would be worth looking at doej's own user association under the account? A GrpTRES there would raise the same reason code, and scontrol show job won't say which association actually tripped it. The tree view will show both the account and the user rows: sacctmgr show assoc where account=dedicated-cpu@igroup format=account,parentname,user,grptres Entirely possible you've been through all this already, feel free to ignore if so! Best, Luis Silva On Mon, Aug 10, 2026 at 2:03 PM Rémy Dernat UM via slurm-users < slurm-users@lists.schedmd.com> wrote:

Hi Laura,

Thanks for your suggestion. Unfortunately, no, this account does not have any child account.

Best regards

Le 10 août 2026 18:37:56 GMT+02:00, Laura Hild <lsh@jlab.org> a écrit :

...

I don't suppose dedicated-cpu@igroup has any children? I can only imagine Slurm has a different idea of what the totals are than squeue_account_count_running_rss does. Or maybe you can tell us that over the past week you found some other (weirder?) problem. ------------------------------ Od: Rémy Dernat via slurm-users <slurm-users@lists.schedmd.com> Poslano: torek, 4. avgust 2026 03:28 Za: slurm-users@lists.schedmd.com Zadeva: [slurm-users] Job stuck in pending state with AssocGrpMemLimit

Hi !

I have a very strange behavior that I am unable to explain.

A user have a job stuck in pending state, although there are enough resources to run :

#########################################

[dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools [dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 8949801_[0] cpu-dedic access_d doej PD 0:00 1 (AssocGrpMemLimit) 8936077 cpu-dedic sim11.13 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930985 cpu-dedic sim33.23 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930981 cpu-dedic sim32.14 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930980 cpu-dedic sim31.61 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930979 cpu-dedic sim23.21 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930976 cpu-dedic sim22.27 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930972 cpu-dedic sim21.23 gamma PD 0:00 2 (AssocGrpCpuLimit) 8930971 cpu-dedic sim13.24 gamma PD 0:00 2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315 1420G omega 1294 6 8880518 5000M gamma 1326 191 [dernatr@io-login-01 ~]$ scontrol show job 8949801_0 JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9 UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null) Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53 AccrueTime=2026-08-03T16:59:53 StartTime=Unknown EndTime=Unknown Deadline=N/A SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082 ReqNodeList=(null) ExcNodeList=(null) NodeList= NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* ReqTRES=cpu=1,mem=560G,node=1,billing=1 AllocTRES=(null) Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0 Features=(null) DelayBoot=00:00:00 OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null) Command=/scratch/users/doej/run_access_d49_23.sh WorkDir=/scratch/users/doej AdminComment=159722b1-908e-47e2-82b1-08872ee64906 StdErr=/scratch/users/doej/slurm-8949801_0.err StdIn=/dev/null StdOut=/scratch/users/doej/slurm-8949801_0.out TresPerTask=cpu=1 [dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup Account Org Descr GrpTRES Share MaxWall Def QOS QOS ------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- dedicated-cpu@igroup igroup not_defined cpu=364,mem=3360G 1 dedicated dedicate [dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem AccountingStorageTRES = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb DefMemPerCPU = 2048 MaxMemPerNode = UNLIMITED PropagateResourceLimitsExcept = MEMLOCK SelectTypeParameters = CR_CORE_MEMORY MemorySwappiness = (null) [dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn dedicated-cpu@igroup doej 1 0.005495 782016 0.003362 0.335501 dedicated-cpu@igroup omega 1 0.005495 2239436 0.009627 0.335119 dedicated-cpu@igroup lambda 1 0.005495 48189472 0.207149 0.334353 dedicated-cpu@igroup gamma 1 0.005495 85232776 0.366384 0.333971 dedicated-cpu@igroup 1 0.030303 232632330 0.129448 Account User RawShares NormShares RawUsage EffectvUsage FairShare -------------------- ---------- ---------- ----------- ----------- ------------- ----------

#########################################

As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB.

I also look at backfilling and fair sharing, but I don't see anything that could block him.

Any idea may be useful,

Thanks,

Best regards,

R

-- ISDM Lundi/Mardi/Jeudi/Vendredi ISEM Mercredi

-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com