[slurm-users] From an initial installation cannot start slurmctld with a slurmdbd running

[Christopher Samuel]

Hi Lawrence,

On 5/17/23 3:26 pm, Sorrillo, Lawrence wrote:

> Here is the error I get:
> 
> slurmctld: fatal: Can not recover assoc_usage state, incompatible 
> version, got 9728 need >= 8704 <= 9216,
> 
> The slurm version is:  20.11.9

That error seems to appear when slurmctld is loading usage data from an 
on-disk cache of data (the "assoc_usage" file) - the function that 
throws that error is called here:

         /* Now load the usage from a flat file since it isn't kept in
            the database
         */
         load_assoc_usage();

It's telling you that the data file was written with a version of Slurm 
ahead of where it's at.

With my little cheat sheet:

 > ./slurmver
SLURM_23_02_PROTOCOL_VERSION = 9984
SLURM_22_05_PROTOCOL_VERSION = 9728
SLURM_21_08_PROTOCOL_VERSION = 9472
SLURM_20_11_PROTOCOL_VERSION = 9216
SLURM_20_02_PROTOCOL_VERSION = 8960
SLURM_19_05_PROTOCOL_VERSION = 8704
SLURM_18_08_PROTOCOL_VERSION = 8448

That tells us the data file was written by Slurm 22.05.x, so my guess is 
that version was tested and the "assoc_usage" file that's being read 
here wasn't cleaned up afterwards.

Hope that helps!

All the best,
Chris
-- 
Chris Samuel  :  http://www.csamuel.org/  :  Berkeley, CA, USA