[slurm-users] SLURM: reconfig

[Ole Holm Nielsen]

On 5/5/22 16:08, Mark Dixon wrote:
> On Thu, 5 May 2022, Ole Holm Nielsen wrote:
> ...
>> That is correct.  Just do "scontrol reconfig" on the slurmctld server.  If
>> all your slurmd's are truly running Configless[1], they will pick up the
>> new config and reconfigure without restarting.
>>
>> Details are summarized in
>> https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#reconfiguration-of-slurm-conf. 
>>
>> Beware that you can't add or remove nodes without restarting.  Also,
>> changing certain slurm.conf parameters require restarting.
> ...
> 
> However...
> 
> Given that the normal recommendation for adding/removing nodes safely is to:
> 
> * stop slurmctld
> * edit slurm.conf etc.
> * restart the slurmd nodes to pick up new slurm.conf
> * start slurmctld
> 
> I'm confused how this is supposed to be achieved in a configless setting, 
> as slurmctld isn't running to distribute the updated files to slurmd.

You're right, probably the correct order for Configless must be:

* stop slurmctld
* edit slurm.conf etc.
* start slurmctld
* restart the slurmd nodes to pick up new slurm.conf

See also slides 29-34 in 
https://slurm.schedmd.com/SLUG21/Field_Notes_5.pdf from the Slurm 
publications site https://slurm.schedmd.com/publications.html

Less-Safe, but usually okay, procedure:
1. Change configs
2. Restart slurmctld
3. Restart all slurmd processes really quickly


/Ole