Hi Xaver, I had the same issue a lot until I got a hint that this is often related to NFS issues. Here is the issue how I see it: If a job is stuck in a system call (like a read() waiting for a hard-mounted NFS to complete) it - as far as I can tell - does not listen to process signals. So SIGTERM (sent by scancel) times out. After KillWait seconds (https://slurm.schedmd.com/slurm.conf.html#OPT_KillWait), Slurm also sends SIGKILL. If that fails, after UnkillableStepTimeout seconds, the entire node is drained with "Kill Task Failed". If/when the job becomes resposive again and terminates, the node is not resumed however, so manual intervention is needed. If you increase the UnkillableStepTimeout parameter in slurm.conf, you might be able to partially resolve this. I hope I could help a tiny bit as I had this exact problem as well. Best wishes, Pierre On 24.08.26 10:46, Xaver Stiensmeier via slurm-users wrote:
Dear slurm-user list,
currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed").
Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue:
[2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18
My *cgroup.conf*:
CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100
From *slurm.conf*:
ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup
*Slurm Version: *slurm 24.11.3
As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas!
Best, Xaver
-- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele@eva.mpg.de Phone: +49 (0) 341 3550 245