[slurm-users] Cleanup of job_container/tmpfs

[Ole Holm Nielsen]

Hi Jason,

IMHO, the job_container/tmpfs is not working well in Slurm 22.05, but 
there may be some significant improvements included in 23.02 (announced 
yesterday).  I've documented our experiences in the Wiki page
https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#temporary-job-directories
This page contains links to bug reports against the job_container/tmpfs 
plugin.

We're using the auto_tmpdir SPANK plugin with great success in Slurm 22.05.

Best regards,
Ole


On 01-03-2023 03:27, Jason Ellul wrote:
> We have recently moved to slurm 22.05.8 and have configured 
> job_container/tmpfs to allow private tmp folders.
> 
> job_container.conf contains:
> 
> AutoBasePath=true
> 
> BasePath=/slurm
> 
> And in slurm.conf we have set
> 
> JobContainerType=job_container/tmpfs
> 
> I can see the folders being created and they are being used but when a 
> job completes the root folder is not being cleaned up.
> 
> Example of running job:
> 
> [root at papr-res-compute204 ~]# ls -al /slurm/14292874
> 
> total 32
> 
> drwx------   3 root      root    34 Mar  1 13:16 .
> 
> drwxr-xr-x 518 root      root 16384 Mar  1 13:16 ..
> 
> drwx------   2 mzethoven root     6 Mar  1 13:16 .14292874
> 
> -r--r--r--   1 root      root     0 Mar  1 13:16 .ns
> 
> Example once job completes /slurm/<jobid> remains:
> 
> [root at papr-res-compute204 ~]# ls -al /slurm/14292794
> 
> total 32
> 
> drwx------   2 root root     6 Mar  1 09:33 .
> 
> drwxr-xr-x 518 root root 16384 Mar  1 13:16 ..
> 
> Is this to be expected or should the folder /slurm/<jobid> also be removed?
> 
> Do I need to create an epilog script to remove the directory that is left?