We are getting a few calls to support container workloads on our Slurm cluster; I want to support these user's usecases, so am looking into it now.
The problem for me is, I'm not super-familiar with container runtimes excepting (regular rootful) Docker... I see that any Slurm-compatible runtime must be "rootless", so that takes out the integration with regular Docker. How does one choose between the many options (from what I gather, Apptainer/Singularity, Podman, and rootless Docker seem to be the most widely-used.) We also have to support GPUs in these containers, but I assume that any of these can support that.
Can anyone tell me what the most popular Slurm-compatible container runtime may be, and plusses/minuses of the three I mentioned above? Thanks!