<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">This sounds similar to something I recently experienced and finally figured out in 21.08.<div class=""><br class=""></div><div class=""><a href="https://lists.schedmd.com/pipermail/slurm-users/2023-January/009594.html" class="">https://lists.schedmd.com/pipermail/slurm-users/2023-January/009594.html</a></div><div class=""><br class=""></div><div class="">The long and short of it, is that I had jobs with the clock running, even though they weren’t showing up in squeue, etc.</div><div class="">I ended up requeueing the jobs, and then cancelling them, and they finally fell off the ledger.</div><div class=""><br class=""></div><div class="">Hope thats helpful,</div><div class="">Reed </div><div class=""></div><div style=""><br class=""><blockquote type="cite" class=""><div class="">On Feb 3, 2023, at 9:17 AM, Jonathan Casco <<a href="mailto:jcasco@fiu.edu" class="">jcasco@fiu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">Hello,<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">We are using Slurm 22.05.6 and have encountered a strange issue with one users jobs where they submitted a job array. The jobs failed and left the queue in the logs but have continued to use CPU minutes well past the job completion. I am using one step as an example here but this is occurring for all the steps within job array.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">Below is a snippet from the slurmctld log for one of the job steps in question:<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-01-25T08:36:40.299] sched/backfill: _start_job: Started JobId=8853669_3(8853785) in <partition> on <node><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-01-25T08:36:40.599] _job_complete: JobId=8853669_3(8853785) WEXITSTATUS 1<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-01-25T08:36:40.601] _job_complete: JobId=8853669_3(8853785) done<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">However when checking the job with sacct I see that the end time is Unknown and the job shows as never completed.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""># sacct -j 8853669_3 --format=start%15,end%15,elapsed%20,state%15<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""> Start End Elapsed State <o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">--------------- --------------- -------------------- --------------- <o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">2023-01-25T08:3 Unknown 9-01:22:21 FAILED <o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""> <o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">One curious bit in this is that the job ID does not appear in the logs of the node where it is said to have run.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">An scancel of the job does not have an effect and we see the following in the logs when attempting to do so:<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-02-03T08:44:36.072] _slurm_rpc_kill_job: REQUEST_KILL_JOB JobId=8853669_3 uid <id><o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-02-03T08:44:36.073] job_str_signal(5): invalid JobId=8853669_3<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">[2023-02-03T08:44:36.073] _slurm_rpc_kill_job: job_str_signal() uid=<id> JobId=8853669_3 sig=9 returned: Invalid job id specified<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">Checking the database everything looks correct there for the job.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">> select time_start,time_end from job_table where id_job="8853669_3";<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">+------------+------------+<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">| time_start | time_end |<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">+------------+------------+<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">| 1674653930 | 1674653931 |<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">+------------+------------+<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">Both slurmctld and slurmdbd are running so I am at a bit of a loss on how to proceed with getting this job to “end” to the controller so that it can stop consuming cpuminutes.<o:p class=""></o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in; font-size: 12pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 11pt;" class="">Any help would be appreciated, thanks!</span></div></div></div></blockquote></div><br class=""></body></html>