I was working on adding some partitions to our test cluster to test preemption. This is Slurm 24.11.1, we have a single slurmctld and slurmdbd in this cluster.
I’m using partition based preemption, not QOS based preemption. I added a QOS to use with the preempting partition to enforce per-job resource limits, and then added the new partitions to slurm.conf. It took me many tries to get the slurm.conf correct, so many attempts to start slurmctld that failed due to conf file parsing errors.
However, once I got the slurm.conf correct, slurmctld would not start, complaining that the cluster ID in the saved state information was wrong:
[2026-07-02T14:55:38.573] fatal: CLUSTER ID MISMATCH.
slurmctld has been started with "ClusterID=3013" from the state files in StateSaveLocation, but the DBD thinks it should be "3391".
Running multiple clusters from a shared StateSaveLocation WILL CAUSE CORRUPTION.
Remove /export/slurm/state/clustername to override this safety check if this is intentional.
The clustername file in the save state location shows the cluster ID 3013. In the Slurm DB, in clusters_table, I see the row for my test cluster shows the (wrong) ID that appears in the logs, 3391, and the mod_time field has a timestamp in the same time frame as when I was doing the work on adding the QOS and partition configuration.
Any thoughts on what could have caused the Slurm DB to be updated with a different cluster ID than what is saved in state? I’d like to avoid something similar happening when I go to deploy the new partitions in production.
John
