[slurm-users] Hi-prio jobs are bypassed by low-prio jobs

[Michał Kadlof]

Hi,

A few tasks with higher priority give way to tasks with lower priority, 
and I don't understand why.

I noticed that the hi-prio tasks require 4 or 8 x GPUs on a single node, 
while the bypassing tasks only use 1 x GPU, but I'm not sure if it's 
related. High-priority tasks have a specific value in the StartTime 
field, but regularly, this value is pushed back to a later time. It 
seems like after finishing a 1GPU task, Slurm immediately schedules 
another 1GPU task instead of waiting for the release of the remaining 3 
or 7 GPUs for a higher-priority task. What can be wrong?

The tasks are being launched in the 'long' partition with QoS named long.

PartitionName=long
    AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
    AllocNodes=ALL Default=NO QoS=long
    DefaultTime=2-00:00:00 DisableRootJobs=NO ExclusiveUser=NO 
GraceTime=0 Hidden=NO
    MaxNodes=UNLIMITED MaxTime=10-00:00:00 MinNodes=0 LLN=NO 
MaxCPUsPerNode=UNLIMITED
    Nodes=dgx-[1-4],sr-[1-3]
    PriorityJobFactor=1 PriorityTier=10000 RootOnly=NO ReqResv=NO 
OverSubscribe=NO
    OverTimeLimit=NONE PreemptMode=SUSPEND
    State=UP TotalCPUs=656 TotalNodes=7 SelectTypeParameters=NONE
    JobDefaults=(null)
    DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
TRES=cpu=656,mem=8255731M,node=7,billing=3474,gres/gpu=32,gres/gpu:a100=32
    TRESBillingWeights=CPU=1,Mem=0.062G,GRES/gpu=72.458

   Name                     GrpTRES
------ ---------------------------
normal
   long  cpu=450,gres/gpu=28,mem=5T


Example of bypassed job with obscured sensitive data:

$ scontrol show job 649800
JobId=649800 JobName=----train with motif
    UserId=XXXXXX(XXXX) GroupId=XXXXXX(XXXXX) MCS_label=N/A
    Priority=275000 Nice=0 Account=sfglab QOS=normal
    JobState=PENDING Reason=QOSGrpGRES Dependency=(null)
    Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
    RunTime=00:00:00 TimeLimit=03:00:00 TimeMin=N/A
    SubmitTime=2023-04-24T12:09:19 EligibleTime=2023-04-24T12:09:19
    AccrueTime=2023-04-24T12:09:19
    StartTime=2023-05-11T06:30:00 EndTime=2023-05-11T09:30:00 Deadline=N/A
    SuspendTime=None SecsPreSuspend=0 LastSchedEval=2023-05-09T13:16:46 
Scheduler=Backfill:*
    Partition=long AllocNode:Sid=0.0.0.0:379113
    ReqNodeList=(null) ExcNodeList=(null)
    NodeList=
    NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16 ReqB:S:C:T=0:0:*:*
    TRES=cpu=16,mem=32G,node=1,billing=16,gres/gpu=8
    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
    MinCPUsNode=16 MinMemoryNode=32G MinTmpDiskNode=0
    Features=(null) DelayBoot=00:00:00
    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=/XXXXX
    WorkDir=/XXXXX
    StdErr=/XXXXXX
    StdIn=/dev/null
    StdOut=/XXXXX
    Power=
    TresPerNode=gres:gpu:8

-- 
best regards | pozdrawiam serdecznie
*Michał Kadlof*
Head of the high performance computing center 	Kierownik ośrodka 
obliczeniowego HPC
Eden^N cluster administrator 	Administrator klastra obliczeniowego Eden^N
Structural and Functional Genomics Laboratory 	Laboratorium Genomiki 
Strukturalnej i Funkcjonalnej
Faculty of Mathematics and Computer Science 	Wydział Matematyki i Nauk 
Informacyjnych
Warsaw University of Technology 	Politechnika Warszawska
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230509/7b7a7fb8/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20230509/7b7a7fb8/attachment.bin>