<div dir="ltr"><div><div dir="ltr"><div>Slurmdbd has an issue and from the logs is still trying to load the old version:<br></div><div><font face="monospace">[2021-01-22T14:17:18.430] MySQL server version is: 5.5.68-MariaDB<br>[2021-01-22T14:17:18.433] error: Database settings not recommended values: innodb_buffer_pool_size innodb_log_file_size innodb_lock_wait_timeout<br>[2021-01-22T14:17:18.528] Accounting storage MYSQL plugin loaded<br>[2021-01-22T14:17:18.529] error: chdir(/var/log): Permission denied<br>[2021-01-22T14:17:18.529] chdir to /var/tmp<br><b>[2021-01-22T14:17:18.531] slurmdbd version 20.02.3 started<br>[2021-01-22T14:56:40.334] error: g_slurm_auth_unpack: remote plugin_id 144 not found</b><br>[2021-01-22T14:56:40.334] error: slurm_unpack_received_msg: Invalid Protocol Version 9216 from uid=-1 from problem connection: Socket operation on non-socket<br>[2021-01-22T14:56:40.334]<b> error: slurm_unpack_received_msg: Incompatible versions of client and server code</b><br>[2021-01-22T14:56:40.345] error: CONN:7 Failed to unpack SLURM_PERSIST_INIT message<br>[2021-03-03T09:49:57.607] Terminate signal (SIGINT or SIGTERM) received<br>[2021-03-03T09:49:57.610] Unable to remove pidfile '/var/run/slurmdbd.pid': Permission denied</font><br></div></div><div><br></div><div>But I know it's updated:</div><div><font face="monospace">rpm -qa|grep slurmdbd<br>slurm20-slurmdbd-20.11.3-mybuild.x86_64</font><br></div><div><br></div><div>And the pid file is not there:</div><div><font face="monospace">ls -l /var/run/slurmdbd.pid<br>ls: cannot access /var/run/slurmdbd.pid: No such file or directory</font><br></div><div><br></div><div>And on the service file:</div><div><font face="monospace">cat /usr/lib/systemd/system/slurmdbd.service<br>[Unit]<br>RequiresMountsFor=/cm/shared<br>Description=Slurm DBD accounting daemon<br>After=network.target munge.service<br>ConditionPathExists=/etc/slurm/slurmdbd.conf<br><br>[Service]<br>Type=simple<br>EnvironmentFile=-/etc/sysconfig/slurmdbd<br><b>ExecStart=/cm/shared/apps/slurm/20.11.3/sbin/slurmdbd -D $SLURMDBD_OPTIONS</b><br>ExecReload=/bin/kill -HUP $MAINPID<br>LimitNOFILE=65536</font><br><br>I reinstalled the slurmdbd file that is local:</div><div><font face="monospace">Dependencies Resolved<br><br>==================================================================================================================================================================================================<br> Package                                    Arch                             Version                                     Repository                                                          Size<br>==================================================================================================================================================================================================<br>Reinstalling:<br> slurm20-slurmdbd                           x86_64                           20.11.3-mybuild                             /slurm20-slurmdbd-20.11.3-mybuild.x86_64                           2.3 M<br><br>Transaction Summary<br>==================================================================================================================================================================================================<br>Reinstall  1 Package<br><br>Total size: 2.3 M<br>Installed size: 2.3 M<br>Is this ok [y/d/N]: y<br>Downloading packages:<br>Running transaction check<br>Running transaction test<br>Transaction test succeeded<br>Running transaction<br>  Installing : slurm20-slurmdbd-20.11.3-mybuild.x86_64                                                                                                                                        1/1<br>  Verifying  : slurm20-slurmdbd-20.11.3-mybuild.x86_64                                                                                                                                        1/1<br><br>Installed:<br>  slurm20-slurmdbd.x86_64 0:20.11.3-mybuild</font><br></div><div><br></div><div>What did I miss? In the <a href="https://slurm.schedmd.com/quickstart_admin.html">upgrade page</a> I see this:</div></div><div><font face="monospace">The libslurm.so version is increased every major release. So things like MPI libraries with Slurm integration should be recompiled. Sometimes it works to just symlink the old .so name(s) to the new one, but this has no guarantee of working.</font><br></div><div><br></div><div>So I have this:</div><div><font face="monospace">locate libslurm.so<br>/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so<br>/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36<br>/cm/shared/apps/slurm/20.11.3/lib64/libslurm.so.36.0.0</font><br></div><div><br></div><div>Is there some other place the old version is being referenced?</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>