[slurm-users] Slurm upgrade to 20.11.3, slurmdbd still trying to start old version 20.02.3

[Robert Kudyba]

Slurmdbd has an issue and from the logs is still trying to load the old
version:
[2021-01-22T14:17:18.430] MySQL server version is: 5.5.68-MariaDB
[2021-01-22T14:17:18.433] error: Database settings not recommended values:
innodb_buffer_pool_size innodb_log_file_size innodb_lock_wait_timeout
[2021-01-22T14:17:18.528] Accounting storage MYSQL plugin loaded
[2021-01-22T14:17:18.529] error: chdir(/var/log): Permission denied
[2021-01-22T14:17:18.529] chdir to /var/tmp

*[2021-01-22T14:17:18.531] slurmdbd version 20.02.3
started[2021-01-22T14:56:40.334] error: g_slurm_auth_unpack: remote
plugin_id 144 not found*
[2021-01-22T14:56:40.334] error: slurm_unpack_received_msg: Invalid
Protocol Version 9216 from uid=-1 from problem connection: Socket operation
on non-socket
[2021-01-22T14:56:40.334]* error: slurm_unpack_received_msg: Incompatible
versions of client and server code*
[2021-01-22T14:56:40.345] error: CONN:7 Failed to unpack SLURM_PERSIST_INIT
message
[2021-03-03T09:49:57.607] Terminate signal (SIGINT or SIGTERM) received
[2021-03-03T09:49:57.610] Unable to remove pidfile '/var/run/slurmdbd.pid':
Permission denied

But I know it's updated:
rpm -qa|grep slurmdbd
slurm20-slurmdbd-20.11.3-mybuild.x86_64

And the pid file is not there:
ls -l /var/run/slurmdbd.pid
ls: cannot access /var/run/slurmdbd.pid: No such file or directory

And on the service file:
cat /usr/lib/systemd/system/slurmdbd.service
[Unit]
RequiresMountsFor=/cm/shared
Description=Slurm DBD accounting daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurmdbd.conf

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmdbd
*ExecStart=/cm/shared/apps/slurm/20.11.3/sbin/slurmdbd -D $SLURMDBD_OPTIONS*
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536

I reinstalled the slurmdbd file that is local:
Dependencies Resolved

==================================================================================================================================================================================================
 Package                                    Arch
  Version                                     Repository
                                       Size
==================================================================================================================================================================================================
Reinstalling:
 slurm20-slurmdbd                           x86_64
  20.11.3-mybuild
/slurm20-slurmdbd-20.11.3-mybuild.x86_64                           2.3 M

Transaction Summary
==================================================================================================================================================================================================
Reinstall  1 Package

Total size: 2.3 M
Installed size: 2.3 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : slurm20-slurmdbd-20.11.3-mybuild.x86_64

                                       1/1
  Verifying  : slurm20-slurmdbd-20.11.3-mybuild.x86_64

                                       1/1

Installed:
  slurm20-slurmdbd.x86_64 0:20.11.3-mybuild

What did I miss? In the upgrade page
<https://slurm.schedmd.com/quickstart_admin.html> I see this:
The libslurm.so version is increased every major release. So things like
MPI libraries with Slurm integration should be recompiled. Sometimes it
works to just symlink the old .so name(s) to the new one, but this has no
guarantee of working.

So I have this:
locate libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36
/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36.0.0

Is there some other place the old version is being referenced?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.schedmd.com/pipermail/slurm-users/attachments/20210303/de0a29f3/attachment.htm>