Good afternoon,
we're still running our HPC cluster on Debian 12 Bookworm, which comes with Slurm 22.05.8 - no issues so far. Yesterday, another machine was added as a submit node (slurm-client only), with Debian 13 Trixie / Slurm 24.11.5. Now on that the commands (sinfo etc.) don't work and the slurmctld logs "Incompatible versions of client and server code". Since upgrading the whole cluster is not an option yet: Has someone attempted (and possibly succeeded) to build 22.05.8 (or 23.11.4? which IIRC would be the last compatible one) for Trixie, and would be willing to share ideas?
Thanks, Steffen
Hi Steffen,
that error is expected with a Slurm 24.11 client talking to a 22.05 controller — the RPC/protocol versions are simply too far apart. SchedMD generally recommends keeping Slurm components reasonably close in version, and upgrading the controller + client commands together (or controller first).
If upgrading the cluster isn’t an option yet, the most practical workaround is to run matching 22.05.x client tools on the Debian 13 submit node (e.g., build/package Slurm 22.05.8 from source for Trixie, or use a small Bookworm chroot/container with slurm-client). In general, I’d expect you’ll need client commands that match the controller’s major version as long as slurmctld stays on 22.05.
Best regards, Rikka Göring (GenericRikka) Maintainer, sysutils/slurm-wlm FreeBSD HPC Modernization Project (unofficial) https://kavocado.net
________________________________ Von: Steffen Grunewald via slurm-users slurm-users@lists.schedmd.com Gesendet: Freitag, 12. Dezember 2025 14:02 An: Slurm users slurm-users@lists.schedmd.com Betreff: [slurm-users] Build 22.05.8 for Debian Trixie?
Good afternoon,
we're still running our HPC cluster on Debian 12 Bookworm, which comes with Slurm 22.05.8 - no issues so far. Yesterday, another machine was added as a submit node (slurm-client only), with Debian 13 Trixie / Slurm 24.11.5. Now on that the commands (sinfo etc.) don't work and the slurmctld logs "Incompatible versions of client and server code". Since upgrading the whole cluster is not an option yet: Has someone attempted (and possibly succeeded) to build 22.05.8 (or 23.11.4? which IIRC would be the last compatible one) for Trixie, and would be willing to share ideas?
Thanks, Steffen
-- Steffen Grunewald, Cluster Administrator Max Planck Institute for Gravitational Physics (Albert Einstein Institute) Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany ~~~ Fon: +49-331-567 7274 Mail: steffen.grunewald(at)aei.mpg.de ~~~
-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com
Hi,
On Fri, 2025-12-12 at 14:16:25 +0000, Slurm users wrote:
Hi Steffen,
that error is expected with a Slurm 24.11 client talking to a 22.05 controller
that's why I'm looking for some 23.11 which in theory would be inside the compatibility window...
If upgrading the cluster isn�t an option yet, the most practical workaround
is to run matching 22.05.x client tools on the Debian 13 submit node (e.g., build/package Slurm 22.05.8 from source for Trixie
This is where I'm constantly failing, for multiple reasons: the *.so libraries are installed 0555 instead of 0755 which they had been set before, perl versions aren't properly detected (perl/5.40.1 vs perl5/5.40) etc.
, or use a small Bookworm chroot/container with slurm-client)
This indeed might make sense but is too user-unfriendly, I'm afraid.
. In general, I�d expect you�ll need client commands that match the controller�s major version as long as slurmctld stays on 22.05.
For the theory part above, I found some 23.11.4 backport attempt but even that one throws an error "slurm_load_partitions: Unexpected message received" with server- side "Invalid Protocol Version 10240" (the 24.11 one was 10752, btw).
So this makes me believe that the server version must be >= the client one, despite windows usually being transparent from both sides - and I must find a way to build 22.05 slurm-client, slurm-wlm-basic-plugins and slurm-wlm-jwt-plugin packages for Trixie; or have to wait until the controller can be upgraded. (I hope to keep the DB; there's an intermediate 23.11.7-1 build that might still fit trixie for this. That's a different story though.)
Maybe I'd "just" build the binaries within Trixie? (I'm so used to packages, and packaging, but this is a single point of failure...) In theory that should work, and I don't need everything now...
Thanks, S
On 12/15/25 11:14 am, Steffen Grunewald via slurm-users wrote:
that's why I'm looking for some 23.11 which in theory would be inside the compatibility window...
Problem is that for versions:
compute <= slurmctld <= slurmdbd
so your compute nodes need to be the same as (or 2 versions earlier) than slurmctld - but not any later version.
Good morning,
On Mon, 2025-12-15 at 12:18:02 -0500, Slurm users wrote:
On 12/15/25 11:14 am, Steffen Grunewald via slurm-users wrote:
that's why I'm looking for some 23.11 which in theory would be inside the compatibility window...
Problem is that for versions:
compute <= slurmctld <= slurmdbd
so your compute nodes need to be the same as (or 2 versions earlier) than slurmctld - but not any later version.
Thanks, that clarifies things that aren't expressed as clearly in the context of the compat window itself - the information although is there in the upgrade order instructions.
Since we were upgrading from Bookworm, we put the 22.05 packages on hold and kept all available fingers crossed, and it seems that Trixie can live with the old packages.
Whether we'll be equally lucky when upgrading the controller node (we need to use 23.11 as an intermediate, and aren't quite sure whether we should run that in Bookworm or Trixie) - time will have to tell.
Thanks so far,
Steffen
-
Christopher Samuel -
Rikka Göring -
Steffen Grunewald