Ciao Diego,
I find it extremely hard to understand situations like this. I wish Slurm were more clear on how it reported what it is doing, but I digress...

I suspect that there are other job(s) which have higher priority than this one which are supposed to run on that node but cannot start because maybe this/these high-priority jobs(s) need(s) several nodes and the other nodes are not available at the moment?
Pure speculation, obviously, since I have no idea what the rest of your cluster looks like, and what the rest of the workflow is, but the clue/hint is

> JobState=PENDING Reason=Priority Dependency=(null)

You are pending because something else has higher priority. Going back to my first sentence, I wish Slurm would say which one other job (maybe there are more than one, but one would suffice for this investigation) is trumping this job priority so one could more clearly understand what is going on, without sleuthing.

HTH


On Fri, Dec 6, 2024 at 7:36 AM Diego Zuccato via slurm-users <slurm-users@lists.schedmd.com> wrote:
Hello all.
An user reported that a job wasn't starting, so I tried to replicate the
request and I get:
-8<--
[root@ophfe1 root.old]# scontrol show job 113936
JobId=113936 JobName=test.sh
    UserId=root(0) GroupId=root(0) MCS_label=N/A
    Priority=1 Nice=0 Account=root QOS=long
    JobState=PENDING Reason=Priority Dependency=(null)
    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
    RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
    SubmitTime=2024-12-06T13:19:36 EligibleTime=2024-12-06T13:19:36
    AccrueTime=2024-12-06T13:19:36
    StartTime=Unknown EndTime=Unknown Deadline=N/A
    SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-12-06T13:21:32
Scheduler=Backfill:*
    Partition=m3 AllocNode:Sid=ophfe1:855189
    ReqNodeList=(null) ExcNodeList=(null)
    NodeList=
    NumNodes=1-1 NumCPUs=96 NumTasks=96 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
    TRES=cpu=96,mem=95000M,node=1,billing=1296
    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
    MinCPUsNode=1 MinMemoryNode=95000M MinTmpDiskNode=0
    Features=(null) DelayBoot=00:00:00
    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=/home/root.old/test.sh
    WorkDir=/home/root.old
    StdErr=/home/root.old/%N-%J.err
    StdIn=/dev/null
    StdOut=/home/root.old/%N-%J.out
    Power=


[root@ophfe1 root.old]# scontrol sho partition m3
PartitionName=m3
    AllowGroups=ALL DenyAccounts=formazione AllowQos=ALL
    AllocNodes=ALL Default=NO QoS=N/A
    DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO GraceTime=0
Hidden=NO
    MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO
MaxCPUsPerNode=UNLIMITED
    Nodes=mtx20
    PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO
OverSubscribe=NO
    OverTimeLimit=NONE PreemptMode=CANCEL
    State=UP TotalCPUs=192 TotalNodes=1
SelectTypeParameters=CR_SOCKET_MEMORY
    JobDefaults=(null)
    DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
    TRES=cpu=192,mem=1150000M,node=1,billing=2592
    TRESBillingWeights=CPU=13.500,Mem=2.2378G

[root@ophfe1 root.old]# scontrol show node mtx20
NodeName=mtx20 Arch=x86_64 CoresPerSocket=24
    CPUAlloc=0 CPUEfctv=192 CPUTot=192 CPULoad=0.00
    AvailableFeatures=ib,matrix,intel,avx
    ActiveFeatures=ib,matrix,intel,avx
    Gres=(null)
    NodeAddr=mtx20 NodeHostName=mtx20 Version=22.05.6
    OS=Linux 4.18.0-372.9.1.el8.x86_64 #1 SMP Tue May 10 14:48:47 UTC 2022
    RealMemory=1150000 AllocMem=0 FreeMem=1156606 Sockets=4 Boards=1
    MemSpecLimit=2048
    State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=8 Owner=N/A MCS_label=N/A
    Partitions=m3
    BootTime=2024-12-06T10:01:42 SlurmdStartTime=2024-12-06T10:02:54
    LastBusyTime=2024-12-06T10:51:58
    CfgTRES=cpu=192,mem=1150000M,billing=2592
    AllocTRES=
    CapWatts=n/a
    CurrentWatts=0 AveWatts=0
    ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

-8<--

So the node is free, the partition does not impose extra limits (used
only for accounting factors) but the job does not start.

Any hints?

Tks

--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786


--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com