I am trying to integrate Rootless Docker with Slurm. have set-up Rootless Docker as per the docs "https://slurm.schedmd.com/containers.html" . I have scrum.lua, oci.conf (for crun) and slurm.conf in place. Then "~/.config/docker/daemon.json" and "~/.config/systemd/user/docker.service.d/override.conf" are in place too. But I can't seem to get it to work:

$ docker run $DOCKER_SECURITY alpine /bin/printenv SLURM_JOB_ID
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4abcf2066143: Pull complete
Digest: sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
Status: Downloaded newer image for alpine:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/user/1000/docker-exec/containerd/daemon/io.containerd.runtime.v2.task/moby/97e8dd767977ac03ab7af54c015c0fd5dfd26e737771b977acb7e41f799023aa/log.json: no such file or directory): /usr/bin/scrun did not terminate successfully: exit status 1: unknown.

One thing is if I don't use Slurm as the runtime for Docker (if I remove the " ~/.config/docker/daemon.json") then docker runs fine