Hello,

Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: 

ls -lsrt /scratch
drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe

The epilog script usually takes care of deleting it: 
echo -e "\n**** JOB EPILOG ****\n"

if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then
    rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID}
fi

Sometimes, the node get in draining state because of an epilog error for : 
[2026-08-12T16:36:30.215] error: epilog failed: rc:1 output:
**** JOB EPILOG ****
rm: cannot remove '/scratch/johndoe/464668': Permission denied

This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? 

Best,

Fritz Ratnasamy
Data Scientist
Information Technology

--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com