Hey Pierre, your explanation would fit well as our NFS definitely has to work a lot currently. We do have increased KillWait slightly to address this issue, but maybe that was just not enough. We will try this! If anyone has a solution that really solves the underlying issue, that would be wonderful of course! Best, Xaver On 8/24/26 12:18, Pierre Abele via slurm-users wrote:

Hi Xaver,

I had the same issue a lot until I got a hint that this is often related to NFS issues.

Here is the issue how I see it: If a job is stuck in a system call (like a read() waiting for a hard-mounted NFS to complete) it - as far as I can tell - does not listen to process signals. So SIGTERM (sent by scancel) times out. After KillWait seconds (https://slurm.schedmd.com/slurm.conf.html#OPT_KillWait), Slurm also sends SIGKILL. If that fails, after UnkillableStepTimeout seconds, the entire node is drained with "Kill Task Failed".

If/when the job becomes resposive again and terminates, the node is not resumed however, so manual intervention is needed.

If you increase the UnkillableStepTimeout parameter in slurm.conf, you might be able to partially resolve this.

I hope I could help a tiny bit as I had this exact problem as well.

Best wishes, Pierre

On 24.08.26 10:46, Xaver Stiensmeier via slurm-users wrote:

...

Dear slurm-user list,

currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed").

Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue:

    [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2:     cgroup_p_step_destroy: still unable to remove job's step cgroup (/     sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or     resource busy     [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup:     proctrack_p_signal: sending process 160348 (slurm_task) signal 15     [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup:     proctrack_p_signal: sending process 160672 (slurm_task) signal 18     [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2:     cgroup_p_step_destroy: still unable to remove job's step cgroup (/     sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or     resource busy     [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup:     proctrack_p_signal: sending process 160672 (slurm_task) signal 15     [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup:     proctrack_p_signal: sending process 166993 (slurm_task) signal 18

My *cgroup.conf*:

    CgroupMountpoint="/sys/fs/cgroup"     ConstrainCores=no     ConstrainRAMSpace=yes     ConstrainSwapSpace=no     ConstrainDevices=no     AllowedRamSpace=100     AllowedSwapSpace=0     MaxRAMPercent=100     MaxSwapPercent=100

 From *slurm.conf*:

    ProctrackType=proctrack/cgroup     TaskPlugin=task/cgroup

*Slurm Version: *slurm 24.11.3

As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas!

Best, Xaver

Hi, We’ve encountered similar issues with Nextflow and Snakemake job cancelations and have been increasing our UnkillableStepTimeout to help avoid draining nodes. We’re currently at 7min: UnkillableStepTimeout = 420 sec 7min isn’t a silver bullet number. I just wanted to reinforce that tuning UnkillableStepTimeout is a viable option, and that you may need to add significant time (especially for I/O to complete). Good luck! — Sebastian Smith Principal Systems Engineer | High-Performance Computing sebastian.smith@seattlechildrens.org Cell (360) 358-3283 SeattleChildrens.org [A black and grey logo Description automatically generated] From: Pierre Abele via slurm-users <slurm-users@lists.schedmd.com> Date: Monday, August 24, 2026 at 6:28 AM To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com> Subject: [slurm-users] Re: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup If it helps, these are our settings, running Slurm 25.05.8: # scontrol show config | grep -i unkillable UnkillableStepProgram = (null) UnkillableStepTimeout = 300 sec # scontrol show config | grep -i killwait KillWait = 30 sec KillWait is actually at its default, while UnkillableStepTimeout is increased to 5 minutes (default is 1 minute). To determine if your root cause is NFS, you could monitor for high values of CPU-iowait in the compute nodes. If these correlate with unkillable jobs, you might have an indicator. Best, Pierre On 24.08.26 13:47, Xaver Stiensmeier via slurm-users wrote:

Hey Pierre,

your explanation would fit well as our NFS definitely has to work a lot currently. We do have increased KillWait slightly to address this issue, but maybe that was just not enough. We will try this!

If anyone has a solution that really solves the underlying issue, that would be wonderful of course!

Best, Xaver

On 8/24/26 12:18, Pierre Abele via slurm-users wrote:

...

Hi Xaver,

I had the same issue a lot until I got a hint that this is often related to NFS issues.

Here is the issue how I see it: If a job is stuck in a system call (like a read() waiting for a hard-mounted NFS to complete) it - as far as I can tell - does not listen to process signals. So SIGTERM (sent by scancel) times out. After KillWait seconds (https:// slurm.schedmd.com/slurm.conf.html#OPT_KillWait), Slurm also sends SIGKILL. If that fails, after UnkillableStepTimeout seconds, the entire node is drained with "Kill Task Failed".

If/when the job becomes resposive again and terminates, the node is not resumed however, so manual intervention is needed.

If you increase the UnkillableStepTimeout parameter in slurm.conf, you might be able to partially resolve this.

I hope I could help a tiny bit as I had this exact problem as well.

Best wishes, Pierre

On 24.08.26 10:46, Xaver Stiensmeier via slurm-users wrote:

...

Dear slurm-user list,

currently, I am seeing the same issue again and again; maybe like once a week. Where when a user stops a nextflow run, terminating many jobs at the same time, the system puts nodes in drain saying they did not stop in time ("drained" "Kill task failed").

Looking at the *slurmd* log files indicate that not stopping the job, but removing the job's step cgroup is the issue:

[2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or resource busy [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160348 (slurm_task) signal 15 [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 18 [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup (/ sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or resource busy [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 160672 (slurm_task) signal 15 [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup: proctrack_p_signal: sending process 166993 (slurm_task) signal 18

My *cgroup.conf*:

CgroupMountpoint="/sys/fs/cgroup" ConstrainCores=no ConstrainRAMSpace=yes ConstrainSwapSpace=no ConstrainDevices=no AllowedRamSpace=100 AllowedSwapSpace=0 MaxRAMPercent=100 MaxSwapPercent=100

From *slurm.conf*:

ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup

*Slurm Version: *slurm 24.11.3

As I often recognize this issue after it occurs, I cannot really debug during the removal attempt and I am unsure how to debug this issue further. I am thankful for any ideas!

Best, Xaver

-- Pierre Abele, M.Sc. HPC Administrator Max-Planck-Institute for Evolutionary Anthropology Department of Primate Behavior and Evolution Deutscher Platz 6 04103 Leipzig Room: U2.80 E-Mail: pierre_abele@eva.mpg.de Phone: +49 (0) 341 3550 245 CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.