Hi Michael,

We're on RHEL 9 - it's a newly commissioned system without any (real) users yet, so we have the relative freedom to make fairly substantial changes without impacting any production work for the moment.

I've tried various combinations of storage.conf settings (we already set runroot to a similar /tmp location and graphroot is to an NFS-mounted user home so that the user image library persists across all nodes).... but I always found Podman throw an error relating to creating an 'events' folder under /run/user/$UID .. And I just couldn't figure out which setting this was (setting the podman event backend type to 'none' stopped the mkdir error, but also seemed to prevent Podman from running).

It sounds like the prolog/epilog solution is going to be the easiest route to resolve this.

I hadn't thought about a clean up script for those /tmp entries... but yeah, that's clearly going to need to be put in place as well.

Thanks for the ideas!

John 
From: Paul Edmon via slurm-users <slurm-users@lists.schedmd.com>
Sent: 05 September 2025 14:29
To: slurm-users@lists.schedmd.com <slurm-users@lists.schedmd.com>
Subject: [slurm-users] Re: Creating /run/user/$UID - for Podman runtime
 
⚠ External sender. Take care when opening links or attachments. Do not provide your login details.

We recently setup the same thing (Rocky 8). What we did was we set
/etc/containers/storage.conf and pointed the following variables to /tmp:

storage.conf:runroot = "/tmp/containers-user-$UID/storage"
storage.conf:graphroot = "/tmp/containers-user-$UID/storage"
storage.conf:rootless_storage_path = "/tmp/containers-user-$UID/storage"

We also have a prune script which cleans up /tmp periodically keeping it
clean.

I like your solution for subuid, we put together a puppet module that
does much the same thing: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffasrc%2Fpuppet-subuid&data=05%7C02%7Cjohn.snowdon%40newcastle.ac.uk%7Cd8b416396e1944caff5208ddec80c62d%7C9c5012c9b61644c2a91766814fbe3e87%7C1%7C0%7C638926760013036340%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=tsnBt3D8O5d2IEjK%2BJXPc8i4P%2FSkjqNsuaJAaZ0OkLI%3D&reserved=0

-Paul Edmon-

On 9/5/25 9:20 AM, Michael DiDomenico via slurm-users wrote:
> for what it's worth, we found the simplest solution was just to run a
> prolog/epilog to create the directories and clean them up.  it's only
> a couple lines of bash.
>
> On Fri, Sep 5, 2025 at 7:59 AM John Snowdon via slurm-users
> <slurm-users@lists.schedmd.com> wrote:
>> We are in the middle of implementing an extensive range of container support on our new HPC platform and have decided to offer our users a wide suite of technologies to better support their workloads:
>>
>> Apptainer
>> Podman (rootless)
>> Docker (rootless)
>>
>>
>> We've already got a solution for automated entries in /etc/subuid and /etc/subgid on the head nodes (available here under GPL: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmegatron-uk%2Fpam_subid&data=05%7C02%7Cjohn.snowdon%40newcastle.ac.uk%7Cd8b416396e1944caff5208ddec80c62d%7C9c5012c9b61644c2a91766814fbe3e87%7C1%7C0%7C638926760013055007%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZFO4mbu14EWbpYYJLdBl3nJGjzlU7yp1UN97XH3sN0Q%3D&reserved=0), which is where we intend users to build their container images, and building and running containers using Apptainer and Podman in those environments works really well - we're happy that it should take care of 95% of our users needs (Docker is the last few percent....) and not involve giving them any special permissions.
>>
>> If I ssh directly to a compute node, then Podman also works there to run an existing image (podman container run ...).
>>
>> What I'm struggling with now is running Podman under Slurm itself on our compute nodes.
>>
>> It appears as though Podman (in rootless mode) wants to put the majority of its run time / state information under /run/user/$UID ... this is fine on the head nodes which have interactive logins hitting PAM modules which instantiate the /run/user/$UID directories, but not under sbatch/srun which doesn't create them by default.
>>
>> I've not been able to find a single, magical setting which will move all of the Podman state information out from /run/user to another location - there are 3 or 4 settings involved, and even then I still find various bits of Podman want to create stuff under there.
>>
>> Rather than hacking away at getting Podman changed to move all settings and state information elsewhere, it seems like the cleanest solution would just be to put the regular /run/user/$UID directory in place at the point Slurm starts the job instead.
>>
>> What's the best way to get Slurm to create this and clean-up afterwards? Should this be in a prolog/epilog wrapper (e.g. directly calling loginctl) or is it cleaner to get Slurm to trigger the usual PAM session machinery in some manner?
>>
>> John Snowdon
>> Senior Research Infrastructure Engineer (HPC)
>>
>> Research Software Engineering
>> Catalyst Building, Room 2.01
>> Newcastle University
>> 3 Science Square
>> Newcastle Helix
>> Newcastle upon Tyne
>> NE4 5TG
>> https://hpc.researchcomputing.ncl.ac.uk
>>
>> --
>> slurm-users mailing list -- slurm-users@lists.schedmd.com
>> To unsubscribe send an email to slurm-users-leave@lists.schedmd.com

--
slurm-users mailing list -- slurm-users@lists.schedmd.com
To unsubscribe send an email to slurm-users-leave@lists.schedmd.com