Timed-out jobs were 6.6% of jobs but ~45% of job energy on two DOE clusters
Hello, I've been working through job-scheduler records from two DOE clusters (Eagle and Kestrel) and one result seems worth putting in front of this list. Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of total job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing. The part I didn't expect is how predictable they are before they start. Four summary statistics (mean, standard deviation, range, mean absolute successive difference) over a user's previous 24 Elapsed/Timelimit ratios, plain logistic regression, using nothing at all about the job being predicted. Forward-chained by month, never trained on anything later than the month being scored. The prediction figures are Kestrel only: about 6.8M job windows over 22 monthly splits. At an 80th-percentile threshold, pooled across splits, 95.4% of timed-out jobs were flagged, at the cost of flagging 16.9% of jobs that would have finished fine. Per split that is 93.2% +/- 6.3 and 21.3% +/- 12.5. The false-flag spread is wide, and it is the number that decides whether this is usable at a given site, so I would look at that before the headline. The model is deliberately boring, and that is the point. What took the work was the measurement around it: forward-chaining by month so nothing is ever trained on the future, a label-shuffle control that has to land at chance before any number is printed, and a refusal path so it prints nothing at all when the data won't carry a conclusion. 868 lines total, so you can audit the lot in an afternoon. I packaged it so you can run it against your own sacct export: git clone https://github.com/JJardine919/lattice24-assess cd lattice24-assess && python3 -m venv .venv && .venv/bin/pip install -e . sacct --allocations --parsable2 --starttime=$(date -d '12 months ago' +%F) \ -o User,End,Timelimit,Elapsed,State,ConsumedEnergyRaw > export.psv .venv/bin/lattice24-assess export.psv --out ./report If your site already has numpy and scikit-learn available, skip the venv and run python3 -m lattice24_assess.cli export.psv --out ./report from the clone. It runs entirely on the login node. There are no network calls in it, and the source is short enough to confirm that yourself in a couple of minutes. User IDs are hashed with a per-run salt; job names, working directories and command lines are dropped at parse time and never read. It refuses rather than degrades: under 6 months of data, under 50k windows, under 200 timeout events, or if the label-shuffle control doesn't land near chance, it explains why and prints no headline number. Two things it deliberately will not tell you. It reports energy AT STAKE, not energy saved -- a flag avoids nothing until somebody resizes, fixes or declines the job, and that multiplier belongs to your site, not to me. And it covers TIMEOUT only; CANCELLED and FAILED are excluded, so your recoverable waste is probably larger than what it reports. Repo, Apache-2.0: https://github.com/JJardine919/lattice24-assess Method and results: https://doi.org/10.5281/zenodo.21911351 I'd genuinely like to know whether the pattern holds elsewhere. If your number comes out much lower, that is a useful result and I'd like to hear it. Jim Jardine Lattice24
On 9/21/26 12:07 pm, James Jardine via slurm-users wrote:
Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of total job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing.
That's not necessarily true, it ran to the timelimit true, but that might be the users intention, they could be doing regular checkpoints and then restarting from them to carry on their simulation/calculation. We see that a lot. So I would caution against drawing that sort of conclusion without checking with the users running those jobs. All the best, Chris
Chris, that's a fair point, and I overstated it. A job that checkpoints, runs to its limit and resubmits has done useful work, so counting all of its energy as waste is wrong. The tool reports energy "at stake" rather than energy saved partly for this reason, but saying the job "returns nothing" went too far. I think the two cases can be told apart from accounting data alone. A checkpoint-restart chain usually looks like the same user resubmitting a near-identical job shortly after the timeout, and a genuine failure usually doesn't. I'll add that split, so the report shows TIMEOUT energy both with and without likely restart chains instead of a single number. If you have a rough idea of what fraction of timeouts at your site are intentional, that would help me calibrate it. Thanks for the correction. Jim On Mon, 21 Sept 2026 at 12:03, Christopher Samuel via slurm-users < slurm-users@lists.schedmd.com> wrote:
On 9/21/26 12:07 pm, James Jardine via slurm-users wrote:
Jobs that ended in TIMEOUT were about 6.6% of jobs but roughly 45% of
total
job energy. That follows once you say it out loud: a timed-out job runs its full wall-clock request by definition, and then returns nothing.
That's not necessarily true, it ran to the timelimit true, but that might be the users intention, they could be doing regular checkpoints and then restarting from them to carry on their simulation/calculation. We see that a lot.
So I would caution against drawing that sort of conclusion without checking with the users running those jobs.
All the best, Chris
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Chris, Following up on your point about checkpoint/restart. The split is in lattice24-assess 0.2.0: https://github.com/JJardine919/lattice24-assess/commit/54041cb A TIMEOUT is now marked as a likely restart link when the same user starts another job with the same time limit within --restart-gap hours (default 2). The report gives timeout energy, and energy at stake, both with and without those links, instead of a single number. It is still a proxy, since the tool never reads job names, so it will miss chains that change their limit or sit in the queue longer than the gap, and it will occasionally catch unrelated back-to-back jobs. On synthetic data it found every planted chain with no false matches. On data with no chains it marked about 1.5% of timeouts. The report says to read the two figures as a range. I've also taken "returns nothing" out of the README and the report. If your site's pattern doesn't fit the same-limit, short-gap shape (for example, restarts that come back with a different limit), I'd like to know, because that is where the proxy would undercount. Thanks again, Jim
participants (2)
-
Christopher Samuel -
James Jardine