Our cluster has developed a strange intermittent behaviour where jobs are being put into a pending state because they aren't passing the AssocGrpCpuLimit, even though the user submitting has enough cpus for the job to run.
For example:
$ squeue -o "%.6i %.9P %.8j %.8u %.2t %.10M %.7m %.7c %.20R" JOBID PARTITION NAME USER ST TIME MIN_MEM MIN_CPU NODELIST(REASON) 799 normal hostname andrewss PD 0:00 2G 5 (AssocGrpCpuLimit)
..so the job isn't running, and it's the only job in the queue, but:
$ sacctmgr list associations part=normal user=andrewss format=Account,User,Partition,Share,GrpTRES Account User Partition Share GrpTRES ---------- ---------- ---------- --------- ------------- andrewss andrewss normal 1 cpu=5
That user has a limit of 5 CPUs so the job should run.
The weird thing is that this effect is intermittent. A job can hang and the queue will stall for ages but will then suddenly start working and you can submit several jobs and they all work, until one fails again.
The cluster has active nodes and plenty of resource:
$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST normal* up infinite 2 idle compute-0-[6-7] interactive up 1-12:00:00 3 idle compute-1-[0-1,3]
The slurmctld log just says:
[2024-03-14T16:21:41.275] _slurm_rpc_submit_batch_job: JobId=799 InitPrio=4294901720 usec=259
Whilst it's in this state I can run other jobs with core requests of up to 4 and they work, but not 5. It's like slurm is adding one CPU to the request and then denying it.
I'm sure I'm missing something fundamental but would appreciate it if someone could point out what it is!
Thanks
Simon.