I don't know. Maybe you could test the existence and permissions then log these before the deletion ? Are you not better to use TMPDIR and define which filesystem TMPDIR uses? Are the failed jobs very short? Is this some caching weirdness with a parallel filesystem On Thu, Aug 13, 2026, 6:39 AM Ratnasamy, Fritz via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hello,
Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe
The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n"
if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi
Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied
This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening?
Best,
*Fritz Ratnasamy*Data Scientist Information Technology
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com