If it helps, these are our settings, running Slurm 25.05.8: # scontrol show config | grep -i unkillable UnkillableStepProgram = (null) UnkillableStepTimeout = 300 sec # scontrol show config | grep -i killwait KillWait = 30 sec KillWait is actually at its default, while UnkillableStepTimeout is increased to 5 minutes (default is 1 minute). To determine if your root cause is NFS, you could monitor for high values of CPU-iowait in the compute nodes. If these correlate with unkillable jobs, you might have an indicator. Best, Pierre On 24.08.26 13:47, Xaver Stiensmeier via slurm-users wrote:
Hey Pierre,
your explanation would fit well as our NFS definitely has to work a lot currently. We do have increased KillWait slightly to address this issue, but maybe that was just not enough. We will try this!
If anyone has a solution that really solves the underlying issue, that would be wonderful of course!
Best, Xaver
On 8/24/26 12:18, Pierre Abele via slurm-users wrote:
Hi Xaver,
I had the same issue a lot until I got a hint that this is often related to NFS issues.
Here is the issue how I see it: If a job is stuck in a system call (like a read() waiting for a hard-mounted NFS to complete) it - as far as I can tell - does not listen to process signals. So SIGTERM (sent by scancel) times out. After KillWait seconds (https:// slurm.schedmd.com/slurm.conf.html#OPT_KillWait), Slurm also sends SIGKILL. If that fails, after UnkillableStepTimeout seconds, the entire node is drained with "Kill Task Failed".
If/when the job becomes resposive again and terminates, the node is not resumed however, so manual intervention is needed.
If you increase the UnkillableStepTimeout parameter in slurm.conf, you might be able to partially resolve this.
I hope I could help a tiny bit as I had this exact problem as well.
Best wishes, Pierre
On 24.08.26 10:46, Xaver Stiensmeier via slurm-users wrote:
Dear slurm-user list,
currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed").
Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue:
[2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18
My *cgroup.conf*:
CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100
From *slurm.conf*:
ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup
*Slurm Version: *slurm 24.11.3
As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas!
Best, Xaver
-- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele@eva.mpg.de Phone: +49 (0) 341 3550 245