Hi,

We’ve encountered similar issues with Nextflow and Snakemake job cancelations and have been increasing our UnkillableStepTimeout to help avoid draining nodes.  We’re currently at 7min:

UnkillableStepTimeout   = 420 sec

7min isn’t a silver bullet number.  I just wanted to reinforce that tuning UnkillableStepTimeout is a viable option, and that you may need to add significant time (especially for I/O to complete).

Good luck!

—
Sebastian Smith
Principal Systems Engineer | High-Performance Computing
sebastian.smith@seattlechildrens.org
Cell         (360) 358-3283
SeattleChildrens.org
A black and grey logo Description automatically generated

From: Pierre Abele via slurm-users <slurm-users@lists.schedmd.com>
Date: Monday, August 24, 2026 at 6:28 AM
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Re: cgroup/v2: cgroup_p_step_destroy: still unable to remove job's step cgroup

If it helps, these are our settings, running Slurm 25.05.8:

# scontrol show config | grep -i unkillable
UnkillableStepProgram   = (null)
UnkillableStepTimeout   = 300 sec
# scontrol show config | grep -i killwait
KillWait                = 30 sec

KillWait is actually at its default, while UnkillableStepTimeout is
increased to 5 minutes (default is 1 minute).

To determine if your root cause is NFS, you could monitor for high
values of CPU-iowait in the compute nodes. If these correlate with
unkillable jobs, you might have an indicator.

Best,
Pierre

On 24.08.26 13:47, Xaver Stiensmeier via slurm-users wrote:
> Hey Pierre,
>
> your explanation would fit well as our NFS definitely has to work a lot
> currently. We do have increased KillWait slightly to address this issue,
> but maybe that was just not enough. We will try this!
>
> If anyone has a solution that really solves the underlying issue, that
> would be wonderful of course!
>
> Best,
> Xaver
>
> On 8/24/26 12:18, Pierre Abele via slurm-users wrote:
>> Hi Xaver,
>>
>> I had the same issue a lot until I got a hint that this is often
>> related to NFS issues.
>>
>> Here is the issue how I see it: If a job is stuck in a system call
>> (like a read() waiting for a hard-mounted NFS to complete) it - as far
>> as I can tell - does not listen to process signals. So SIGTERM (sent
>> by scancel) times out. After KillWait seconds (https://
>> slurm.schedmd.com/slurm.conf.html#OPT_KillWait), Slurm also sends
>> SIGKILL. If that fails, after UnkillableStepTimeout seconds, the
>> entire node is drained with "Kill Task Failed".
>>
>> If/when the job becomes resposive again and terminates, the node is
>> not resumed however, so manual intervention is needed.
>>
>> If you increase the UnkillableStepTimeout parameter in slurm.conf, you
>> might be able to partially resolve this.
>>
>> I hope I could help a tiny bit as I had this exact problem as well.
>>
>> Best wishes,
>> Pierre
>>
>>
>> On 24.08.26 10:46, Xaver Stiensmeier via slurm-users wrote:
>>> Dear slurm-user list,
>>>
>>> currently, I am seeing the same issue again and again; maybe like
>>> once a week. Where when a user stops a nextflow run, terminating many
>>> jobs at the same time, the system puts nodes in drain saying they did
>>> not stop in time ("drained" "Kill task failed").
>>>
>>> Looking at the *slurmd* log files indicate that not stopping the job,
>>> but removing the job's step cgroup is the issue:
>>>
>>>     [2026-08-18T14:12:44.729] [251978.batch] debug2: cgroup/v2:
>>>     cgroup_p_step_destroy: still unable to remove job's step cgroup (/
>>>     sys/fs/cgroup/system.slice/slurmstepd.scope/job_251978): Device or
>>>     resource busy
>>>     [2026-08-18T14:12:44.730] [251978.extern] debug2: proctrack/cgroup:
>>>     proctrack_p_signal: sending process 160348 (slurm_task) signal 15
>>>     [2026-08-18T14:12:49.197] [251982.extern] debug2: proctrack/cgroup:
>>>     proctrack_p_signal: sending process 160672 (slurm_task) signal 18
>>>     [2026-08-18T14:12:49.216] [251982.batch] debug2: cgroup/v2:
>>>     cgroup_p_step_destroy: still unable to remove job's step cgroup (/
>>>     sys/fs/cgroup/system.slice/slurmstepd.scope/job_251982): Device or
>>>     resource busy
>>>     [2026-08-18T14:12:49.217] [251982.extern] debug2: proctrack/cgroup:
>>>     proctrack_p_signal: sending process 160672 (slurm_task) signal 15
>>>     [2026-08-18T15:53:27.057] [252037.extern] debug2: proctrack/cgroup:
>>>     proctrack_p_signal: sending process 166993 (slurm_task) signal 18
>>>
>>> My *cgroup.conf*:
>>>
>>>     CgroupMountpoint="/sys/fs/cgroup"
>>>     ConstrainCores=no
>>>     ConstrainRAMSpace=yes
>>>     ConstrainSwapSpace=no
>>>     ConstrainDevices=no
>>>     AllowedRamSpace=100
>>>     AllowedSwapSpace=0
>>>     MaxRAMPercent=100
>>>     MaxSwapPercent=100
>>>
>>>  From *slurm.conf*:
>>>
>>>     ProctrackType=proctrack/cgroup
>>>     TaskPlugin=task/cgroup
>>>
>>> *Slurm Version: *slurm 24.11.3
>>>
>>> As I often recognize this issue after it occurs, I cannot really
>>> debug during the removal attempt and I am unsure how to debug this
>>> issue further. I am thankful for any ideas!
>>>
>>> Best,
>>> Xaver
>>>
>>>
>>>
>>
>>
>
>

--
Pierre Abele, M.Sc.

HPC Administrator
Max-Planck-Institute for Evolutionary Anthropology
Department of Primate Behavior and Evolution

Deutscher Platz 6
04103 Leipzig

Room: U2.80
E-Mail: pierre_abele@eva.mpg.de
Phone: +49 (0) 341 3550 245

CONFIDENTIALITY NOTICE: This e-mail, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information protected by law. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.