Hi,

On Fri, 2025-12-12 at 14:16:25 +0000, Slurm users wrote:

Hi Steffen,

that error is expected with a Slurm 24.11 client talking to a 22.05 controller

that's why I'm looking for some 23.11 which in theory would be inside the compatibility window...

If upgrading the cluster isn�t an option yet, the most practical workaround

is to run matching 22.05.x client tools on the Debian 13 submit node (e.g., build/package Slurm 22.05.8 from source for Trixie

This is where I'm constantly failing, for multiple reasons: the *.so libraries are installed 0555 instead of 0755 which they had been set before, perl versions aren't properly detected (perl/5.40.1 vs perl5/5.40) etc.

, or use a small Bookworm chroot/container with slurm-client)

This indeed might make sense but is too user-unfriendly, I'm afraid.

. In general, I�d expect you�ll need client commands that match the controller�s major version as long as slurmctld stays on 22.05.

For the theory part above, I found some 23.11.4 backport attempt but even that one throws an error "slurm_load_partitions: Unexpected message received" with server- side "Invalid Protocol Version 10240" (the 24.11 one was 10752, btw).

So this makes me believe that the server version must be >= the client one, despite windows usually being transparent from both sides - and I must find a way to build 22.05 slurm-client, slurm-wlm-basic-plugins and slurm-wlm-jwt-plugin packages for Trixie; or have to wait until the controller can be upgraded. (I hope to keep the DB; there's an intermediate 23.11.7-1 build that might still fit trixie for this. That's a different story though.)

Maybe I'd "just" build the binaries within Trixie? (I'm so used to packages, and packaging, but this is a single point of failure...) In theory that should work, and I don't need everything now...

Thanks, S

Good morning,

On Mon, 2025-12-15 at 12:18:02 -0500, Slurm users wrote:

On 12/15/25 11:14 am, Steffen Grunewald via slurm-users wrote:

...

that's why I'm looking for some 23.11 which in theory would be inside the compatibility window...

Problem is that for versions:

compute <= slurmctld <= slurmdbd

so your compute nodes need to be the same as (or 2 versions earlier) than slurmctld - but not any later version.

Thanks, that clarifies things that aren't expressed as clearly in the context of the compat window itself - the information although is there in the upgrade order instructions.

Since we were upgrading from Bookworm, we put the 22.05 packages on hold and kept all available fingers crossed, and it seems that Trixie can live with the old packages.

Whether we'll be equally lucky when upgrading the controller node (we need to use 23.11 as an intermediate, and aren't quite sure whether we should run that in Bookworm or Trixie) - time will have to tell.

Thanks so far,

Steffen

On Thu, 2026-01-15 at 10:08:25 +0100, Slurm users wrote:

Beeing late to the party - Just add bookworm sources to your trixie and install the slurm-client from bookworm. Seems to work:

sed -e 's/trixie/bookworm/g' </etc/apt/sources.list.d/debian.sources >/etc/apt/sources.list.d/debian12.sources apt-get update apt-get install slurm-client=22.05.8-4+deb12u3 slurm-wlm-basic-plugins:amd64=22.05.8-4+deb12u3

Indeed that's what we did - although "the other way 'round": we marked the old packages (we also need slurm-wlm-jwt-plugin) as held and upgraded everything else. Magic or not, who knows. We can only proceed with the newer versions once the manager has been dist-upgrade'd, which may take a while longer.

Thanks, S