[slurm-users] Jobs stuck with BeginTime and prolog exit status 99:0

[Chandler]

Could you help me figure out why our jobs are stuck PD because of BeginTime? e.g:

              JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
              24458      defq cromwell smrtanal PD       0:00      1 (BeginTime)

# scontrol show job 24458
JobId=24458 JobName=cromwell_d72d675a_dataset_filter
    UserId=smrtanalysis(1002) GroupId=smrtanalysis(1002) MCS_label=N/A
    Priority=4294892709 Nice=0 Account=(null) QOS=normal
    JobState=PENDING Reason=BeginTime Dependency=(null)
    Requeue=1 Restarts=784 BatchFlag=1 Reboot=0 ExitCode=0:0
    RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
    SubmitTime=2022-05-17T09:23:03 EligibleTime=2022-05-17T09:25:04
    AccrueTime=2022-05-17T09:25:04
    StartTime=2022-05-17T09:25:04 EndTime=Unknown Deadline=N/A
    SuspendTime=None SecsPreSuspend=0 LastSchedEval=2022-05-17T09:23:03
    Partition=defq AllocNode:Sid=EagI:2725352
    ReqNodeList=(null) ExcNodeList=(null)
    NodeList=(null)
    BatchHost=EagI
    NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
    TRES=cpu=1,node=1,billing=1
    Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
    MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
    Features=(null) DelayBoot=00:00:00
    OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
    Command=(null)
    WorkDir=/data2/pacbio/smrtlink/jobs
    StdErr=/data2/pacbio/smrtlink/jobs/cromwell-executions/pb_export_ccs/441e90d6-263b-41a5-bbbb-5009d9a346d9/call-prepare_input/prepare_input/d72d675a-4df7-4e9f-8072-e722742f48e7/call-dataset_filter/execution/stderr
    StdIn=/dev/null
    StdOut=/data2/pacbio/smrtlink/jobs/cromwell-executions/pb_export_ccs/441e90d6-263b-41a5-bbbb-5009d9a346d9/call-prepare_input/prepare_input/d72d675a-4df7-4e9f-8072-e722742f48e7/call-dataset_filter/execution/stdout
    Power=
#

/var/log/slurmctld:
[2022-05-17T09:20:44.366] Requeuing JobId=24458
[2022-05-17T09:23:03.068] backfill: Started JobId=24458 in defq on EagI
[2022-05-17T09:23:03.106] error: prolog_slurmctld JobId=24458 prolog exit status 99:0
[2022-05-17T09:23:03.114] Requeuing JobId=24458

Thanks
-- 
Chandler Sobel-Sorenson (he/him) / Systems Administrator
Arizona Genomics Institute
www.genome.arizona.edu