[slurm-users] SLURM in K8s, any advice?

[Viessmann Hans-Nikolai (PSI)]

Hi Nicolas and Urban,

Thank you for your replies!

Kind regards,
Hans

________________________________________
From: slurm-users <slurm-users-bounces at lists.schedmd.com> on behalf of Urban Borštnik <urban.borstnik at id.ethz.ch>
Sent: 16 November 2022 16:45
To: slurm-users at lists.schedmd.com
Subject: Re: [slurm-users] SLURM in K8s, any advice?

Hi Hans,


We run Slurm in k8s at the ETH Zurich to manage physical compute nodes. The link you include and Nicolas's followup already contain the basics.


We build several Docker containers based on CentOS 7 (for now) with Slurm compiled from source for the following services:

  *   slurmdbd
  *   slurmctld
  *   slurmd (used for testing as “containerized nodes”)

All these containers include an sssd daemon that interfaces with the central LDAP though we are looking at ways to streamline this part.

We use several helper containers, such as mariadb, a prometheus exporter, a file server for the code and configuration (used to transfer these to the physical nodes), and a controller that configures users, accounts, QOS, … into Slurm.


PVCs hosted on an NFS appliance provide data persistence.


A Helm chart is used to for deploying to a local test k8s instance, a test/staging cluster, and the production cluster. The chart and containers are site specific but I am happy to share the relevant code & config with you if you contact me by PM.


With kind regards,

Urban


On 2022-11-14 09:42, Viessmann Hans-Nikolai (PSI) wrote:

Good Morning,

I'm working on a project at work to run SLURM cluster management components
(slurmctld and slurmdbd) as K8s pods, which manage a cluster of physical compute
nodes. I've come upon a few discussions of doing this (or more generally running
SLURM in containers); I especially found this one
(see https://groups.google.com/g/slurm-users/c/uevFWPHHr2U/m/fkwusc0JDwAJ)
very helpful.

Are there any further details or advice anyone has on such a setup?

Thank you and kind regards,
Hans

---------------------------------------------------------------------------------------------
Paul Scherrer Institut
Hans-Nikolai Viessmann
High Performance Computing & Emerging Technologies
Building/Room: OHSA/D02
Forschungsstrasse 111
5232 Villigen PSI
Switzerland

Telephone: +41 56 310 41 24
E-Mail: hans-nikolai.viessmann at psi.ch<mailto:hans-nikolai.viessmann at psi.ch>
GPG: 46F7 826E 80E1 EE45 2DCA 1BFC A39B E4B6 EA0C E4C4


--
ETH Zurich, Dr. Urban Borštnik
High Performance Computing, Scientific IT Services
OCT G35, Binzmühlestrasse 130, 8092 Zurich, Switzerland
Phone +41 44 632 3512, http://www.id.ethz.ch/
urban.borstnik at id.ethz.ch<mailto:urban.borstnik at id.ethz.ch>