Dear All,

We set up scrun (slurm 23.11.5) integrated with rootless podman, referring to the official documentation.
https://slurm.schedmd.com/containers.html#podman-scrun
https://slurm.schedmd.com/scrun.html#SECTION_Example-%3CB%3Escrun.lua%3C/B%3E-scripts

However, runc/crun prints the error message and fails to run the container.
Looking at the error message from runc, it seems that there is a problem with the GID mapping configuration.

- runc
  [hoge@slm-master hoge]$ podman run centos echo "Hello World"
  time="2024-04-03T02:39:36Z" level=error msg="runc run failed: cannot specify gid=5 mount option for rootless container: User namespaces enabled, but no group mapping found."
  srun: error: slm-worker: task 0: Exited with exit code 1

- crun
  [hoge@slm-master hoge]$ podman run centos echo "Hello World"
  srun: error: slm-worker: task 0: Exited with exit code 1
  mount `devpts` to `dev/pts`: Invalid argument

When we removed the process of linux.gidMappings configuration from scrun.lua,
runc failed to run the container due to the different error, but crun succeeded.

- scrun.lua (The process of linux.gidMappings configuration)
... snip ...
  -- Provide default group map as root if one not provided
  -- mappings fail with build???
  if (true or config["linux"]["gidMappings"] == nil)
  then
      config["linux"]["gidMappings"] =
              {{containerID=process_group_id, hostID=math.floor(group_id), size=1}}
  end
... snip ...

- runc
  [hoge@slm-master hoge]$ podman run centos echo "Hello World"
  time="2024-04-03T02:37:18Z" level=error msg="runc run failed: User namespaces enabled, but no gid mappings found."
  srun: error: slm-worker: task 0: Exited with exit code 1

- crun
  [hoge@slm-master hoge]$ podman run centos echo "Hello World"
  Hello World

Therefore, we would like to know the following two questions.
  - The reason for the failure to run the container
  - Required or recommended configuration to run the container

Best Regards,
Toshiki Sonoda