I didn't mention that I have:

SchedulerType=sched/backfill

in slurm.conf

I am reading https://slurm.schedmd.com/sched_config.html where it is written:

If the job under consideration can start immediately without impacting the expected start time of any higher priority job, then it does so

but is is also written something that I am not able to fully understand:


For performance reasons, the backfill scheduler reserves whole nodes for jobs, even if jobs don't require whole nodes


Does this mean that the worker nodes listed in the "squeue --start" output [*] are basically not usable until those jobs will start running ?

This would explain my problem, but I don't understand the logic of this behavior

Thanks again
Massimo


[*]
[sgaravat@cld-ter-ui-01 ~]$ squeue --start
             JOBID PARTITION     NAME     USER ST          START_TIME  NODES SCHEDNODES           NODELIST(REASON)
            283565      gpus vllm-pod ciangott PD 2026-04-14T14:22:38      1 cld-ter-gpu-01       (Resources)
            284080 gpus,gpus qed_supe catalano PD 2026-04-14T21:26:22      1 cld-ter-gpu-05       (Resources)
            284081 gpus,gpus qed_supe catalano PD 2026-04-14T22:32:57      1 cld-ter-gpu-04       (Priority)
            284099 gpus,gpus qed_supe catalano PD 2026-04-15T07:19:36      1 cld-ter-gpu-03       (Priority)
            284119 onlycpus- myscript sgaravat PD 2026-04-15T11:12:26      1 btc-dfa-gpu-02       (Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority partitions)
            284121 onlycpus- myscript sgaravat PD 2026-04-15T11:12:26      1 cld-dfa-gpu-06       (Priority)
            284090 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284091 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284092 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284093 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284094 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284095 gpus,gpus long_isi  pavesic PD                 N/A      1 (null)               (Priority)
            284104 gpus,gpus qed_supe catalano PD                 N/A      1 (null)               (Priority)


On Mon, Apr 13, 2026 at 1:02 PM Massimo Sgaravatto <massimo.sgaravatto@gmail.com> wrote:
Dear all

I (try to) manage a slurm cluster composed by some CPU-only nodes and some worker nodes which have also GPUs:

NodeName=cld-ter-[01-06] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 RealMemory=1536000 State=UNKNOWN
NodeName=cld-ter-gpu-[01-05] Sockets=2 CoresPerSocket=96 ThreadsPerCore=2 Gres=gpu:nvidia-h100:4 RealMemory=1536000 State=UNKNOWN

The GPU nodes are exposed through multiple partitions:


PartitionName=gpus Nodes=cld-ter-gpu-[01-02] State=UP PriorityTier=20
PartitionName=sparch Nodes=cld-ter-gpu-03 AllowAccounts=sparch,operators QoS=sparch State=UP PriorityTier=20
PartitionName=geant4 Nodes=cld-ter-gpu-03 AllowAccounts=geant4,operators QoS=geant4 State=UP PriorityTier=20
PartitionName=enipred Nodes=cld-ter-gpu-04 AllowAccounts=enipred,operators QoS=enipred State=UP PriorityTier=20
PartitionName=enipiml Nodes=cld-ter-gpu-05 AllowAccounts=enipiml,operators QoS=enipiml State=UP PriorityTier=20



We also set a partition to allow cpu-only jobs on the GPU nodes, but these jobs should be preempted (killed and requeued) if jobs submitted to partitions with higher priorities require those resources:



PreemptType=preempt/partition_prio
PreemptMode=REQUEUE
PartitionName=onlycpus-opp Nodes=cld-ter-gpu-[01-05],cld-dfa-gpu-06,btc-dfa-gpu-02 State=UP PriorityTier=10

Now, I don't understand why this job [*] submitted on the onlycpus-opp partition can't start running e.g. on the cld-ter-gpu-01, since it has a lot of free resources:

[sgaravat@cld-ter-ui-01 ~]$ scontrol show node cld-ter-gpu-01
NodeName=cld-ter-gpu-01 Arch=x86_64 CoresPerSocket=96
   CPUAlloc=8 CPUEfctv=384 CPUTot=384 CPULoad=5.93
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=gpu:nvidia-h100:4
   NodeAddr=cld-ter-gpu-01 NodeHostName=cld-ter-gpu-01 Version=25.11.3
   OS=Linux 5.14.0-611.45.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 1 05:56:53 EDT 2026
   RealMemory=1536000 AllocMem=560000 FreeMem=1192357 Sockets=2 Boards=1
   State=MIXED+PLANNED ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=gpus,onlycpus-opp
   BootTime=2026-04-09T10:39:35 SlurmdStartTime=2026-04-09T10:40:01
   LastBusyTime=2026-04-09T11:54:46 ResumeAfterTime=None
   CfgTRES=cpu=384,mem=1500G,billing=839,gres/gpu=4,gres/gpu:nvidia-h100=4
   AllocTRES=cpu=8,mem=560000M,gres/gpu=4,gres/gpu:nvidia-h100=4
   CurrentWatts=0 AveWatts=0


I guess the "MIXED+PLANNED" is the answer, but as far as I can see only a job (283469) is planned  for this worker node:

sgaravat@cld-ter-ui-01 ~]$ squeue --start | grep ter-gpu-01
             JOBID PARTITION     NAME     USER ST          START_TIME  NODES SCHEDNODES           NODELIST(REASON)
           
            283469      gpus vllm-pod ciangott PD 2026-04-13T14:31:40      1 cld-ter-gpu-01       (Resources)
            
But job 283469 doesn't require too many resources [**], so the 2 jobs could run together. Why job 283534 can't start ?
Any hints ?

Thanks, Massimo



[*]

[sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283534
JobId=283534 JobName=myscript.sh
   UserId=sgaravat(5008) GroupId=tbadmin(5001) MCS_label=N/A
   Priority=542954 Nice=0 Account=operators QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:41 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2026-04-13T11:10:13 EligibleTime=2026-04-13T11:10:13
   AccrueTime=2026-04-13T11:10:13
   StartTime=2026-04-13T11:58:39 EndTime=2026-04-14T11:58:39 Deadline=N/A
   PreemptEligibleTime=2026-04-13T11:58:39 PreemptTime=None
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:58:39 Scheduler=Backfill
   Partition=onlycpus-opp AllocNode:Sid=cld-ter-ui-01:3035857
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=btc-dfa-gpu-02
   BatchHost=btc-dfa-gpu-02
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=100G,node=1,billing=26
   AllocTRES=cpu=1,mem=100G,node=1,billing=26
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=100G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
   Command=/shared/home/sgaravat/myscript.sh
   SubmitLine=sbatch myscript.sh
   WorkDir=/shared/home/sgaravat
   StdErr=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.err
   StdIn=/dev/null
   StdOut=/shared/home/sgaravat/JOB-myscript.sh.283534.4294967294.out
   MailUser=massimo.sgaravatto@pd.infn.it MailType=INVALID_DEPEND,BEGIN,END,FAIL,REQUEUE,STAGE_OUT

[**]
sgaravat@cld-ter-ui-01 ~]$ scontrol show job=283469
JobId=283469 JobName=vllm-pod
   UserId=ciangott(6054) GroupId=tbuser(6000) MCS_label=N/A
   Priority=499703 Nice=0 Account=cms QOS=normal
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=1-00:00:00 TimeMin=N/A
   SubmitTime=2026-04-13T06:48:37 EligibleTime=2026-04-13T06:48:37
   AccrueTime=2026-04-13T06:48:37
   StartTime=2026-04-13T14:31:40 EndTime=2026-04-14T14:31:40 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-13T11:59:48 Scheduler=Main
   Partition=gpus AllocNode:Sid=cld-ter-ui-01:3015801
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList= SchedNodeList=cld-ter-gpu-01
   NumNodes=1-1 NumCPUs=32 NumTasks=1 CPUs/Task=32 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=32,mem=190734M,node=1,billing=118,gres/gpu=2,gres/gpu:nvidia-h100=2
   AllocTRES=(null)
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=32 MinMemoryNode=190734M MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
   Command=.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm
   SubmitLine=sbatch .interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.slurm
   WorkDir=/shared/home/ciangott
   StdErr=
   StdIn=/dev/null
   StdOut=/shared/home/ciangott/.interlink/jobs/default-0c0257f8-d1ea-4135-a602-96c229ce8516/job.out
   TresPerNode=gres/gpu:nvidia-h100:2
   TresPerTask=cpu=32