[slurm-users] container on slurm cluster

[Markus Kötter]

Hi,

On 18.05.22 08:25, Stephan Roth wrote:

> Personal note: I'm not sure what I'd choose as a successor to 
> Singularity 3.8, yet. Thoughts are welcome.

I can recommend nvidia enroot/pyxis.
enroot does unprivileged sandboxes/containers, pyxis is the slurm SPANK 
glue.

https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf

https://github.com/NVIDIA/enroot
https://github.com/NVIDIA/pyxis


Notes from operation:
I recommend using nvme for the container storage, default configuration 
uses tmpfs.

> …/enroot.conf

…
> ENROOT_RUNTIME_PATH /tmp/enroot/user-$(id -u)

> ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)




tmpfiles.d/…
> d     /tmp/enroot/ 0777 root root - -
> d     /tmp/enroot-data/ 0777 root root - -
> d     /tmp/pyaxis-runtime/ 0777 root root - -


>     dest: /etc/slurm/plugstack.conf.d/pyxis.conf

>     content: |

>       required /usr/lib/x86_64-linux-gnu/slurm/spank_pyxis.so runtime_path=/tmp/pyxis-runtime/



Time savers:
The container url formatting is … unexpected, # is used as seperator for 
the path in host:port#path/file - and may need to be escaped to avoid 
getting interpreted as comment.

It uses a netrc formatted .credentials file for container registries 
with authentication.
Insert the credentials twice - with and without port.

Can do more than documented. (e.g. #SBATCH --container-image)



MfG
-- 
Markus Kötter, +49 681 870832434
30159 Hannover, Lange Laube 6
Helmholtz Center for Information Security
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5968 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20220518/ecd6fac3/attachment.bin>