We recently setup the same thing (Rocky 8). What we did was we set /etc/containers/storage.conf and pointed the following variables to /tmp:
storage.conf:runroot = "/tmp/containers-user-$UID/storage" storage.conf:graphroot = "/tmp/containers-user-$UID/storage" storage.conf:rootless_storage_path = "/tmp/containers-user-$UID/storage"
We also have a prune script which cleans up /tmp periodically keeping it clean.
I like your solution for subuid, we put together a puppet module that does much the same thing: https://github.com/fasrc/puppet-subuid
-Paul Edmon-
On 9/5/25 9:20 AM, Michael DiDomenico via slurm-users wrote:
for what it's worth, we found the simplest solution was just to run a prolog/epilog to create the directories and clean them up. it's only a couple lines of bash.
On Fri, Sep 5, 2025 at 7:59 AM John Snowdon via slurm-users slurm-users@lists.schedmd.com wrote:
We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads:
Apptainer Podman (rootless) Docker (rootless)
We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://github.com/megatron-uk/pam_subid), which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions.
If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...).
What I'm struggling with now is running Podman under Slurm itself on our compute nodes.
It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default.
I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there.
Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead.
What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner?
John Snowdon Senior Research Infrastructure Engineer (HPC)
Research Software Engineering Catalyst Building, Room 2.01 Newcastle University 3 Science Square Newcastle Helix Newcastle upon Tyne NE4 5TG https://hpc.researchcomputing.ncl.ac.uk
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com