[slurm-users] Questions about adding new nodes to Slurm

[Tina Friedrich]

Hello,

a lot of people already gave very good answer to how to tackle this.

Still, I thought it worth pointing this out - you said 'you need to 
basically shut down slurm, update the slurm.conf file, then restart'. 
That makes it sound like a major operation with lots of prep required.

It's not like that at all. Updating slurm.conf is not a major operation.

There's absolutely no reason to shut things down first & then change the 
file. You can edit the file / ship out a new version (however you like) 
and then restart the daemons.

The daemons do not have to all be restarted simultaneously. It is of no 
consequence if they're running with out-of-sync config files for a bit, 
really. (There's a flag you can set if you want to suppress the warning 
- 'NO_CONF_HASH' debug flag I think).

Restarting the dameons (slurmctld, slurmd, ...) is safe. It does not 
require cluster downtime or anything.

I control slurm.conf using configuration management; the config 
management process restarts the appropriate daemon (slurmctld, slurmd, 
slurmdbd) if the file changed. This certainly never happens at the same 
time; there's splay in that. It doesn't even necessarily happen on the 
controller first, or anything like that.

What I'm trying to get across - I have a feeling this 'updating the 
cluster wide config file' and 'file must be the same on all nodes' is a 
lot less of a procedure (and a lot less strict) than you currently 
imagine it to be :)

Tina

On 27/04/2021 19:35, David Henkemeyer wrote:
> Hello,
> 
> I'm new to Slurm (coming from PBS), and so I will likely have a few 
> questions over the next several weeks, as I work to transition my 
> infrastructure from PBS to Slurm.
> 
> My first question has to do with *_adding nodes to Slurm_*.  According 
> to the FAQ (and other articles I've read), you need to basically shut 
> down slurm, update the slurm.conf file /*on all nodes in the cluster*/, 
> then restart slurm.
> 
> - Why do all nodes need to know about all other nodes?  From what I have 
> read, its Slurm does a checksum comparison of the slurm.conf file across 
> all nodes.  Is this the only reason all nodes need to know about all 
> other nodes?
> - Can I create a symlink that points <sysconfdir>/slurm.conf to a 
> slurm.conf file on an NFS mount point, which is mounted on all the 
> nodes?  This way, I would only need to update a single file, then 
> restart Slurm across the entire cluster.
> - Any additional help/resources for adding/removing nodes to Slurm would 
> be much appreciated.  Perhaps there is a "toolkit" out there to automate 
> some of these operations (which is what I already have for PBS, and will 
> create for Slurm, if something doesn't already exist).
> 
> Thank you all,
> 
> David

-- 
Tina Friedrich, Advanced Research Computing Snr HPC Systems Administrator

Research Computing and Support Services
IT Services, University of Oxford
http://www.arc.ox.ac.uk http://www.it.ox.ac.uk