"Ratnasamy, Fritz via slurm-users" <slurm-users@lists.schedmd.com> writes:
Sometimes, the node get in draining state because of an epilog error for : [2026-08-12T16:36:30.215] error: epilog failed: rc:1 output: **** JOB EPILOG **** rm: cannot remove '/scratch/johndoe/464668': Permission denied
This seems to happen on random jobs and eventually the files get deleted but the node remains in draining state. Does anyone know what is happening?
While I don't know why it happens, in the epilog, you could check the exit status of "rm", and if it wasn't 0, you could do an "ls -la /scratch/johndoe/" or similar and log that to a log file. Then you might find out what the reason is. (We do things like this, and then we have a script that checks for such log files from the epilogs.) Also, if you don't want the node to be drained when this happens, just make sure the epilog exits with exit status 0 whether the "rm" fails or not. -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo