[slurm-users] Re: Configless Slurm Error: failed to fetch remote configs

Hi Xaver,

We have been running Configless Slurm for a number of years, and we're very happy with this setup. I have documented all the detailed configurations we made in this Wiki page, so maybe you want to consult this page:

https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_configuration/#configless-sl...

IHTH, Ole

On 1/16/26 11:11, Xaver Stiensmeier via slurm-users wrote:

Hey everyone,

in the past we set up clusters with configs on each node. Now we want to explore configless. Without changing anything else, we therefore followed: https://slurm.schedmd.com/configless_slurm.html and added 'enable_configless' in the config on the master:

SlurmctldParameters=cloud_dns,idle_on_node_suspend,enable_configless,reconfig_on_restart

and start each worker's slurmd with the conf-server parameter:

# Override systemd service to set conditional path
[Service]
ExecStart=
ExecStart=/usr/sbin/slurmd --conf-server=master

However, this leads to:

slurmd: error: _fetch_child: failed to fetch remote configs: Protocol
authentication error

slurmd: error: _establish_configuration: failed to load configs.
Retrying in 10 seconds.

on the workers and on the master (/var/log/slurm/slurmctld) to:

[2026-01-16T10:00:06.681] error: Munge decode failed: Invalid credential
[2026-01-16T10:00:06.681] auth/munge: _print_cred: ENCODED: Thu Jan 01
00:00:00 1970
[2026-01-16T10:00:06.681] auth/munge: _print_cred: DECODED: Thu Jan 01
00:00:00 1970
[2026-01-16T10:00:06.681] error: slurm_unpack_received_msg:
[[worker]:24295] auth_g_verify: REQUEST_CONFIG has authentication
error: Unspecified error
[2026-01-16T10:00:06.681] error: slurm_unpack_received_msg:
[[worker]:24295] Protocol authentication error

The munge key setup is the same as before so I don't think there is anything wrong with it unless something changes with configless (slurm.conf):

AuthType=auth/munge
CryptoType=crypto/munge
AuthAltTypes=auth/jwt
AuthAltParameters=jwt_key=/etc/slurm/jwt-secret.key

I found https://groups.google.com/g/slurm-users/c/Q7FVkhx-bOs but this seems unrelated as both can talk fine with each other:

worker:~$ nc -zv master 6817
Connection to master (192.168.20.169) 6817 port [tcp/*] succeeded!

I tried adding more "-v" to the slurmd start, but that did not give more information. I am unsure how to debug this further. Somehow I think it must be a munge issue, but I am confused as this part hasn't changed.

Best regards, Xaver