Hi !

I have a very strange behavior that I am unable to explain.

A user have a job stuck in pending state, although there are enough resources to run :

#########################################

[dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools 
[dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup
Number of CPUs used by dedicated-cpu@igroup: 230
Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       8949801_[0] cpu-dedic access_d  doej PD       0:00      1 (AssocGrpMemLimit)
           8936077 cpu-dedic sim11.13 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930985 cpu-dedic sim33.23 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930981 cpu-dedic sim32.14 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930980 cpu-dedic sim31.61 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930979 cpu-dedic sim23.21 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930976 cpu-dedic sim22.27 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930972 cpu-dedic sim21.23 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930971 cpu-dedic sim13.24 gamma PD       0:00      2 (AssocGrpCpuLimit)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C"
JOBID MIN_MEMORY USER UID CPUS
8946435 64G orbital 1064 32
8929322 480G doej 2746 1
8925315 1420G omega 1294 6
8880518 5000M gamma 1326 191
[dernatr@io-login-01 ~]$ scontrol show job 8949801_0
JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9
   UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A
   Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated
   JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null)
   Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
   SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53
   AccrueTime=2026-08-03T16:59:53
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main
   Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=560G,node=1,billing=1
   AllocTRES=(null)
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/scratch/users/doej/run_access_d49_23.sh
   WorkDir=/scratch/users/doej
   AdminComment=159722b1-908e-47e2-82b1-08872ee64906 
   StdErr=/scratch/users/doej/slurm-8949801_0.err
   StdIn=/dev/null
   StdOut=/scratch/users/doej/slurm-8949801_0.out
   TresPerTask=cpu=1
[dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup
                                                Account                  Org                          Descr                        GrpTRES     Share     MaxWall   Def QOS                  QOS 
------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- 
                                    dedicated-cpu@igroup                igroup                    not_defined              cpu=364,mem=3360G         1             dedicated            dedicate
[dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem
AccountingStorageTRES   = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb
DefMemPerCPU            = 2048
MaxMemPerNode           = UNLIMITED
PropagateResourceLimitsExcept = MEMLOCK
SelectTypeParameters    = CR_CORE_MEMORY
MemorySwappiness        = (null)
[dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn
 dedicated-cpu@igroup    doej          1    0.005495      782016      0.003362   0.335501 
 dedicated-cpu@igroup     omega          1    0.005495     2239436      0.009627   0.335119 
 dedicated-cpu@igroup        lambda          1    0.005495    48189472      0.207149   0.334353 
 dedicated-cpu@igroup gamma          1    0.005495    85232776      0.366384   0.333971 
dedicated-cpu@igroup                      1    0.030303   232632330      0.129448            
Account                    User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare 
-------------------- ---------- ---------- ----------- ----------- ------------- ---------- 

#########################################

As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB.

I also look at backfilling and fair sharing, but I don't see anything that could block him.

Any idea may be useful,

Thanks,

Best regards,

R

-- 
ISDM Lundi/Mardi/Jeudi/Vendredi
ISEM Mercredi