Hello, Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n" if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening? Best, *Fritz Ratnasamy*Data Scientist Information Technology
I don't know. Maybe you could test the existence and permissions then log these before the deletion ? Are you not better to use TMPDIR and define which filesystem TMPDIR uses? Are the failed jobs very short? Is this some caching weirdness with a parallel filesystem On Thu, Aug 13, 2026, 6:39 AM Ratnasamy, Fritz via slurm-users < slurm-users@lists.schedmd.com> wrote:
Hello,
Our prolog script usually creates temporary folder under /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} with the following permissions: ls -lsrt /scratch drwx------ 2 johndoe root 0 Aug 12 23:35 johndoe
The epilog script usually takes care of deleting it: echo -e "\n**** JOB EPILOG ****\n"
if [ ! -z "${SLURM_JOB_USER}" ] && [ ! -z "${SLURM_JOB_ID}" ]; then rm -rf /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID} fi
Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied
This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening?
Best,
*Fritz Ratnasamy*Data Scientist Information Technology
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
"Ratnasamy, Fritz via slurm-users" <slurm-users@lists.schedmd.com> writes:
Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied
This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening?
While I don't know why it happens, in the epilog, you could check the exit status of "rm", and if it wasn't 0, you could do an "ls -la /scratch/johndoe/" or similar and log that to a log file. Then you might find out what the reason is. (We do things like this, and then we have a script that checks for such log files from the epilogs.) Also, if you don't want the node to be drained when this happens, just make sure the epilog exits with exit status 0 whether the "rm" fails or not. -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo
participants (3)
-
Bjørn-Helge Mevik -
John Hearns -
Ratnasamy, Fritz