Yes mysql database is running I can update and check, but I guess the update will break a couple of config , I need to check if this is something safe to do even though it is for my homelab but still :)

Did you try to connect database using mysql command?

mysql --user=slurm --password=slurmdbpass  slurm_acct_db

C. Ahmet Mercan

On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote:

Thank you Ahmet, I dont have a firewall active. And because slurmdbd cannot connect to the database I am not able to getting it to be activated through systemctl I will share the output for slurmdbd -D -vvv shortly but overall it is always saying trying to connect to the db and then retries a couple of times and crashes

R.

On Thu, May 30, 2024 at 2:51 AM mercan ahmet.mercan@uhem.itu.edu.tr wrote:

Hi;

Did you check can you connect db with your conf parameters from
head-node:

mysql --user=slurm --password=slurmdbpass  slurm_acct_db

Also, check and stop firewall and selinux, if they are running.

Last, you can stop slurmdbd, then run run terminal with:

slurmdbd -D -vvv

Regards;

C. Ahmet Mercan

On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote:

...

Hi everyone
I am trying to get slurmdbd to run on my local home server but I
am really struggling.
Note : am a novice slurm user
my slurmdbd always times out even though all the details in the
conf file are correct

My log looks like this

[2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded
[2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp
[2024-05-29T20:51:30.088] debug2: ArchiveScript = (null)
[2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null)
[2024-05-29T20:51:30.088] debug2: AuthInfo = (null)
[2024-05-29T20:51:30.088] debug2: AuthType = auth/munge
[2024-05-29T20:51:30.088] debug2: CommitDelay = 0
[2024-05-29T20:51:30.088] debug2: DbdAddr = localhost
[2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null)
[2024-05-29T20:51:30.088] debug2: DbdHost = head-node
[2024-05-29T20:51:30.088] debug2: DbdPort = 7032
[2024-05-29T20:51:30.088] debug2: DebugFlags = (null)
[2024-05-29T20:51:30.088] debug2: DebugLevel = 6
[2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10
[2024-05-29T20:51:30.088] debug2: DefaultQOS = (null)
[2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log
[2024-05-29T20:51:30.088] debug2: MessageTimeout = 100
[2024-05-29T20:51:30.088] debug2: Parameters = (null)
[2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid
[2024-05-29T20:51:30.088] debug2: PluginDir =
/usr/lib/x86_64-linux-gnu/slurm-wlm
[2024-05-29T20:51:30.088] debug2: PrivateData = none
[2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months*
[2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months*
[2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months*
[2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months
[2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months
[2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months
[2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months
[2024-05-29T20:51:30.088] debug2: SlurmUser = root(0)
[2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null)
[2024-05-29T20:51:30.089] debug2: StorageHost = localhost
[2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db
[2024-05-29T20:51:30.089] debug2: StoragePort = 3306
[2024-05-29T20:51:30.089] debug2: StorageType =
accounting_storage/mysql
[2024-05-29T20:51:30.089] debug2: StorageUser = slurm
[2024-05-29T20:51:30.089] debug2: TCPTimeout = 2
[2024-05-29T20:51:30.089] debug2: TrackWCKey = 0
[2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0
[2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection:
request new connection 1
[2024-05-29T20:51:30.089] debug2: Attempting to connect to
localhost:3306
[2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started
[2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29
20:51:30 2024
[2024-05-29T20:51:30.091] debug2: Everything rolled up
[2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM)
received
[2024-05-29T20:51:49.673] debug: rpc_mgr shutting down



my config file looks like this

ArchiveEvents=yes
ArchiveJobs=yes
ArchiveResvs=yes
ArchiveSteps=no
ArchiveSuspend=no
ArchiveTXN=no
ArchiveUsage=no
PurgeEventAfter=1month
PurgeJobAfter=12month
PurgeResvAfter=1month
PurgeStepAfter=1month
PurgeSuspendAfter=1month
PurgeTXNAfter=12month
PurgeUsageAfter=24month
# Authentication info
AuthType=auth/munge
# slurmDBD info
DbdAddr=localhost
DbdHost=head-node
DbdPort=7032
SlurmUser=root
MessageTimeout=100
DebugLevel=5
#DefaultQOS=normal,standby
LogFile=/var/log/slurmdbd.log
PidFile=/run/slurmdbd.pid
#PrivateData=accounts,users,usage,jobs
#TrackWCKey=yes
#
# Database info
StorageType=accounting_storage/mysql
StorageHost=localhost
StoragePort=3306
StoragePass=slurmdbpass
StorageUser=slurm
StorageLoc=slurm_acct_db
I used standard names and passwords to get started and I will
change later

but everytime I try to start slurmdbd.service it crashes and I
have that log that I shared with you

I use these versions

slurmdbd -V
slurm-wlm 19.05.5
mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu
(x86_64) using readline 5.2
Everything else Is working properly except I cannot get slurmdbd
to work and at this point I exhausted all my possible trials :)
looking for some expert insights :)


Any idea what I am doing wrong here ? Also I didn't compile any
slurm package. I used the binary from apt repos

Any help will be appreciated

Cheers

Rad

-- 

You should fix this error, this not a warning. It is an error:

"slurmdbd: error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout"

error. You can find info at slurm documentation:

https://slurm.schedmd.com/accounting.html#slurm-accounting-configuration-bef...

C. Ahmet Mercan

30.05.2024 16:53 tarihinde Radhouane Aniba via slurm-users yazdı:

Yes I can connect to my database using mysql --user=slurm --password=slurmdbpass  slurm_acct_db and there is no firewall blocking mysql after checking the firewall question

ALso here is the output of slurmdbd -D -vvv (note I can only run this as sudo )

sudo slurmdbd -D -vvv slurmdbd: debug: Log file re-opened slurmdbd: debug: Munge authentication plugin loaded slurmdbd: debug2: mysql_connect() called for db slurm_acct_db slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: debug2: innodb_buffer_pool_size: 134217728 slurmdbd: debug2: innodb_log_file_size: 50331648 slurmdbd: debug2: innodb_lock_wait_timeout: 50 slurmdbd: error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout slurmdbd: Accounting storage MYSQL plugin loaded slurmdbd: debug2: ArchiveDir = /tmp slurmdbd: debug2: ArchiveScript = (null) slurmdbd: debug2: AuthAltTypes = (null) slurmdbd: debug2: AuthInfo = (null) slurmdbd: debug2: AuthType = auth/munge slurmdbd: debug2: CommitDelay = 0 slurmdbd: debug2: DbdAddr = localhost slurmdbd: debug2: DbdBackupHost = (null) slurmdbd: debug2: DbdHost = hannibal-hn slurmdbd: debug2: DbdPort = 7032 slurmdbd: debug2: DebugFlags = (null) slurmdbd: debug2: DebugLevel = 6 slurmdbd: debug2: DebugLevelSyslog = 10 slurmdbd: debug2: DefaultQOS = (null) slurmdbd: debug2: LogFile = /var/log/slurmdbd.log slurmdbd: debug2: MessageTimeout = 100 slurmdbd: debug2: Parameters = (null) slurmdbd: debug2: PidFile = /run/slurmdbd.pid slurmdbd: debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm slurmdbd: debug2: PrivateData = none slurmdbd: debug2: PurgeEventAfter = 1 months* slurmdbd: debug2: PurgeJobAfter = 12 months* slurmdbd: debug2: PurgeResvAfter = 1 months* slurmdbd: debug2: PurgeStepAfter = 1 months slurmdbd: debug2: PurgeSuspendAfter = 1 months slurmdbd: debug2: PurgeTXNAfter = 12 months slurmdbd: debug2: PurgeUsageAfter = 24 months slurmdbd: debug2: SlurmUser = root(0) slurmdbd: debug2: StorageBackupHost = (null) slurmdbd: debug2: StorageHost = localhost slurmdbd: debug2: StorageLoc = slurm_acct_db slurmdbd: debug2: StoragePort = 3306 slurmdbd: debug2: StorageType = accounting_storage/mysql slurmdbd: debug2: StorageUser = slurm slurmdbd: debug2: TCPTimeout = 2 slurmdbd: debug2: TrackWCKey = 0 slurmdbd: debug2: TrackSlurmctldDown= 0 slurmdbd: debug2: acct_storage_p_get_connection: request new connection 1 slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: slurmdbd version 19.05.5 started slurmdbd: debug2: running rollup at Thu May 30 13:50:08 2024 slurmdbd: debug2: Everything rolled up

It goes like this for some time and then it crashes with this message

slurmdbd: Terminate signal (SIGINT or SIGTERM) received slurmdbd: debug: rpc_mgr shutting down

On Thu, May 30, 2024 at 8:18 AM mercan ahmet.mercan@uhem.itu.edu.tr wrote:

Did you try to connect database using mysql command?

mysql --user=slurm --password=slurmdbpass slurm_acct_db

C. Ahmet Mercan

On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote:

...

Thank you Ahmet,
I dont have a firewall active.
And because slurmdbd cannot connect to the database I am not able
to getting it to be activated through systemctl I will share the
output for slurmdbd -D -vvv shortly but overall it is always
saying trying to connect to the db and then retries a couple of
times and crashes

R.




On Thu, May 30, 2024 at 2:51 AM mercan
<ahmet.mercan@uhem.itu.edu.tr> wrote:

    Hi;

    Did you check can you connect db with your conf parameters
    from head-node:

    mysql --user=slurm --password=slurmdbpass slurm_acct_db

    Also, check and stop firewall and selinux, if they are running.

    Last, you can stop slurmdbd, then run run terminal with:

    slurmdbd -D -vvv

    Regards;

    C. Ahmet Mercan

    On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote:

...

    Hi everyone
    I am trying to get slurmdbd to run on my local home server
    but I am really struggling.
    Note : am a novice slurm user
    my slurmdbd always times out even though all the details in
    the conf file are correct

    My log looks like this

    [2024-05-29T20:51:30.088] Accounting storage MYSQL plugin
    loaded
    [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp
    [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null)
    [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null)
    [2024-05-29T20:51:30.088] debug2: AuthInfo = (null)
    [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge
    [2024-05-29T20:51:30.088] debug2: CommitDelay = 0
    [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost
    [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null)
    [2024-05-29T20:51:30.088] debug2: DbdHost = head-node
    [2024-05-29T20:51:30.088] debug2: DbdPort = 7032
    [2024-05-29T20:51:30.088] debug2: DebugFlags = (null)
    [2024-05-29T20:51:30.088] debug2: DebugLevel = 6
    [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10
    [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null)
    [2024-05-29T20:51:30.088] debug2: LogFile =
    /var/log/slurmdbd.log
    [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100
    [2024-05-29T20:51:30.088] debug2: Parameters = (null)
    [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid
    [2024-05-29T20:51:30.088] debug2: PluginDir =
    /usr/lib/x86_64-linux-gnu/slurm-wlm
    [2024-05-29T20:51:30.088] debug2: PrivateData = none
    [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months*
    [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months*
    [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months*
    [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months
    [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months
    [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months
    [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months
    [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0)
    [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null)
    [2024-05-29T20:51:30.089] debug2: StorageHost = localhost
    [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db
    [2024-05-29T20:51:30.089] debug2: StoragePort = 3306
    [2024-05-29T20:51:30.089] debug2: StorageType =
    accounting_storage/mysql
    [2024-05-29T20:51:30.089] debug2: StorageUser = slurm
    [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2
    [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0
    [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0
    [2024-05-29T20:51:30.089] debug2:
    acct_storage_p_get_connection: request new connection 1
    [2024-05-29T20:51:30.089] debug2: Attempting to connect to
    localhost:3306
    [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started
    [2024-05-29T20:51:30.090] debug2: running rollup at Wed May
    29 20:51:30 2024
    [2024-05-29T20:51:30.091] debug2: Everything rolled up
    [2024-05-29T20:51:49.673] Terminate signal (SIGINT or
    SIGTERM) received
    [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down



    my config file looks like this

    ArchiveEvents=yes
    ArchiveJobs=yes
    ArchiveResvs=yes
    ArchiveSteps=no
    ArchiveSuspend=no
    ArchiveTXN=no
    ArchiveUsage=no
    PurgeEventAfter=1month
    PurgeJobAfter=12month
    PurgeResvAfter=1month
    PurgeStepAfter=1month
    PurgeSuspendAfter=1month
    PurgeTXNAfter=12month
    PurgeUsageAfter=24month
    # Authentication info
    AuthType=auth/munge
    # slurmDBD info
    DbdAddr=localhost
    DbdHost=head-node
    DbdPort=7032
    SlurmUser=root
    MessageTimeout=100
    DebugLevel=5
    #DefaultQOS=normal,standby
    LogFile=/var/log/slurmdbd.log
    PidFile=/run/slurmdbd.pid
    #PrivateData=accounts,users,usage,jobs
    #TrackWCKey=yes
    #
    # Database info
    StorageType=accounting_storage/mysql
    StorageHost=localhost
    StoragePort=3306
    StoragePass=slurmdbpass
    StorageUser=slurm
    StorageLoc=slurm_acct_db
    I used standard names and passwords to get started and I
    will change later

    but everytime I try to start slurmdbd.service it crashes and
    I have that log that I shared with you

    I use these versions

    slurmdbd -V
    slurm-wlm 19.05.5
    mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu
    (x86_64) using readline 5.2
    Everything else Is working properly except I cannot get
    slurmdbd to work and at this point I exhausted all my
    possible trials :) looking for some expert insights :)


    Any idea what I am doing wrong here ? Also I didn't compile
    any slurm package. I used the binary from apt repos

    Any help will be appreciated

    Cheers

    Rad

    -- 

-- *Rad Aniba, PhD*

Thank you Ahmet and Brian,

Ahmet, which conf in particular slurmdbd is readiugn from, I parsed all the cnf files for mysql and I cannot find the data it is displaying here

slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: debug2: innodb_buffer_pool_size: 134217728 slurmdbd: debug2: innodb_log_file_size: 50331648 slurmdbd: debug2: innodb_lock_wait_timeout: 50 slurmdbd: error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout

sudo tree /etc/mysql/* /etc/mysql/conf.d ├── mysql.cnf └── mysqldump.cnf /etc/mysql/debian.cnf /etc/mysql/debian-start /etc/mysql/FROZEN /etc/mysql/mariadb.cnf /etc/mysql/mariadb.conf.d ├── 50-client.cnf ├── 50-mysql-clients.cnf ├── 50-mysqld_safe.cnf └── 50-server.cnf /etc/mysql/my.cnf /etc/mysql/my.cnf.fallback /etc/mysql/mysql.cnf /etc/mysql/mysql.conf.d ├── mysql.cnf └── mysqld.cnf

On Thu, May 30, 2024 at 12:21 PM Brian Andrus via slurm-users < slurm-users@lists.schedmd.com> wrote:

That SIGTERM message means something is telling slurmdbd to quit.

Check your cron jobs, maintenance scripts, etc. Slurmdbd is being told to shutdown. If you are running in the foreground, a ^C does that. If you run a kill or killall on it, you will get that same message.

Brian Andrus On 5/30/2024 6:53 AM, Radhouane Aniba via slurm-users wrote:

Yes I can connect to my database using mysql --user=slurm --password=slurmdbpass slurm_acct_db and there is no firewall blocking mysql after checking the firewall question

ALso here is the output of slurmdbd -D -vvv (note I can only run this as sudo )

sudo slurmdbd -D -vvv slurmdbd: debug: Log file re-opened slurmdbd: debug: Munge authentication plugin loaded slurmdbd: debug2: mysql_connect() called for db slurm_acct_db slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: debug2: innodb_buffer_pool_size: 134217728 slurmdbd: debug2: innodb_log_file_size: 50331648 slurmdbd: debug2: innodb_lock_wait_timeout: 50 slurmdbd: error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout slurmdbd: Accounting storage MYSQL plugin loaded slurmdbd: debug2: ArchiveDir = /tmp slurmdbd: debug2: ArchiveScript = (null) slurmdbd: debug2: AuthAltTypes = (null) slurmdbd: debug2: AuthInfo = (null) slurmdbd: debug2: AuthType = auth/munge slurmdbd: debug2: CommitDelay = 0 slurmdbd: debug2: DbdAddr = localhost slurmdbd: debug2: DbdBackupHost = (null) slurmdbd: debug2: DbdHost = hannibal-hn slurmdbd: debug2: DbdPort = 7032 slurmdbd: debug2: DebugFlags = (null) slurmdbd: debug2: DebugLevel = 6 slurmdbd: debug2: DebugLevelSyslog = 10 slurmdbd: debug2: DefaultQOS = (null) slurmdbd: debug2: LogFile = /var/log/slurmdbd.log slurmdbd: debug2: MessageTimeout = 100 slurmdbd: debug2: Parameters = (null) slurmdbd: debug2: PidFile = /run/slurmdbd.pid slurmdbd: debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm slurmdbd: debug2: PrivateData = none slurmdbd: debug2: PurgeEventAfter = 1 months* slurmdbd: debug2: PurgeJobAfter = 12 months* slurmdbd: debug2: PurgeResvAfter = 1 months* slurmdbd: debug2: PurgeStepAfter = 1 months slurmdbd: debug2: PurgeSuspendAfter = 1 months slurmdbd: debug2: PurgeTXNAfter = 12 months slurmdbd: debug2: PurgeUsageAfter = 24 months slurmdbd: debug2: SlurmUser = root(0) slurmdbd: debug2: StorageBackupHost = (null) slurmdbd: debug2: StorageHost = localhost slurmdbd: debug2: StorageLoc = slurm_acct_db slurmdbd: debug2: StoragePort = 3306 slurmdbd: debug2: StorageType = accounting_storage/mysql slurmdbd: debug2: StorageUser = slurm slurmdbd: debug2: TCPTimeout = 2 slurmdbd: debug2: TrackWCKey = 0 slurmdbd: debug2: TrackSlurmctldDown= 0 slurmdbd: debug2: acct_storage_p_get_connection: request new connection 1 slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: slurmdbd version 19.05.5 started slurmdbd: debug2: running rollup at Thu May 30 13:50:08 2024 slurmdbd: debug2: Everything rolled up

It goes like this for some time and then it crashes with this message

slurmdbd: Terminate signal (SIGINT or SIGTERM) received slurmdbd: debug: rpc_mgr shutting down

On Thu, May 30, 2024 at 8:18 AM mercan ahmet.mercan@uhem.itu.edu.tr wrote:

...

Did you try to connect database using mysql command?

mysql --user=slurm --password=slurmdbpass slurm_acct_db

C. Ahmet Mercan

On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote:

Thank you Ahmet, I dont have a firewall active. And because slurmdbd cannot connect to the database I am not able to getting it to be activated through systemctl I will share the output for slurmdbd -D -vvv shortly but overall it is always saying trying to connect to the db and then retries a couple of times and crashes

R.

On Thu, May 30, 2024 at 2:51 AM mercan ahmet.mercan@uhem.itu.edu.tr wrote:

...

Hi;

Did you check can you connect db with your conf parameters from head-node:

mysql --user=slurm --password=slurmdbpass slurm_acct_db

Also, check and stop firewall and selinux, if they are running.

Last, you can stop slurmdbd, then run run terminal with:

slurmdbd -D -vvv

Regards;

C. Ahmet Mercan

On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote:

Hi everyone I am trying to get slurmdbd to run on my local home server but I am really struggling. Note : am a novice slurm user my slurmdbd always times out even though all the details in the conf file are correct

My log looks like this

[2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) [2024-05-29T20:51:30.088] debug2: AuthInfo = (null) [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) [2024-05-29T20:51:30.088] debug2: DbdHost = head-node [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 [2024-05-29T20:51:30.088] debug2: Parameters = (null) [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid [2024-05-29T20:51:30.088] debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm [2024-05-29T20:51:30.088] debug2: PrivateData = none [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) [2024-05-29T20:51:30.089] debug2: StorageHost = localhost [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 [2024-05-29T20:51:30.089] debug2: StorageType = accounting_storage/mysql [2024-05-29T20:51:30.089] debug2: StorageUser = slurm [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: request new connection 1 [2024-05-29T20:51:30.089] debug2: Attempting to connect to localhost:3306 [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 20:51:30 2024 [2024-05-29T20:51:30.091] debug2: Everything rolled up [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) received [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down

my config file looks like this

ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month # Authentication info AuthType=auth/munge # slurmDBD info DbdAddr=localhost DbdHost=head-node DbdPort=7032 SlurmUser=root MessageTimeout=100 DebugLevel=5 #DefaultQOS=normal,standby LogFile=/var/log/slurmdbd.log PidFile=/run/slurmdbd.pid #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql StorageHost=localhost StoragePort=3306 StoragePass=slurmdbpass StorageUser=slurm StorageLoc=slurm_acct_db I used standard names and passwords to get started and I will change later

but everytime I try to start slurmdbd.service it crashes and I have that log that I shared with you

I use these versions

slurmdbd -V slurm-wlm 19.05.5 mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Everything else Is working properly except I cannot get slurmdbd to work and at this point I exhausted all my possible trials :) looking for some expert insights :)

Any idea what I am doing wrong here ? Also I didn't compile any slurm package. I used the binary from apt repos

Any help will be appreciated

Cheers

Rad

--

-- *Rad Aniba, PhD*

-- slurm-users mailing list -- slurm-users@lists.schedmd.com To unsubscribe send an email to slurm-users-leave@lists.schedmd.com

It could be systemd doing that. Since slurmdbd is being started with -D, I would verify that slurmdbd.service has Type=simple and not Type=forking. The systemctl status output later in the thread shows systemd starting slurmdbd with -D.

If that's the slurmdbd package from Ubuntu you might find that it's got the opposite config to what you expect.

Ben.

On 30/05/2024 17:18, Brian Andrus via slurm-users wrote: This email was sent to you by someone outside the University. You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

That SIGTERM message means something is telling slurmdbd to quit.

Check your cron jobs, maintenance scripts, etc. Slurmdbd is being told to shutdown. If you are running in the foreground, a ^C does that. If you run a kill or killall on it, you will get that same message.

Brian Andrus

On 5/30/2024 6:53 AM, Radhouane Aniba via slurm-users wrote: Yes I can connect to my database using mysql --user=slurm --password=slurmdbpass slurm_acct_db and there is no firewall blocking mysql after checking the firewall question

ALso here is the output of slurmdbd -D -vvv (note I can only run this as sudo )

sudo slurmdbd -D -vvv slurmdbd: debug: Log file re-opened slurmdbd: debug: Munge authentication plugin loaded slurmdbd: debug2: mysql_connect() called for db slurm_acct_db slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: debug2: innodb_buffer_pool_size: 134217728 slurmdbd: debug2: innodb_log_file_size: 50331648 slurmdbd: debug2: innodb_lock_wait_timeout: 50 slurmdbd: error: Database settings not recommended values: innodb_buffer_pool_size innodb_lock_wait_timeout slurmdbd: Accounting storage MYSQL plugin loaded slurmdbd: debug2: ArchiveDir = /tmp slurmdbd: debug2: ArchiveScript = (null) slurmdbd: debug2: AuthAltTypes = (null) slurmdbd: debug2: AuthInfo = (null) slurmdbd: debug2: AuthType = auth/munge slurmdbd: debug2: CommitDelay = 0 slurmdbd: debug2: DbdAddr = localhost slurmdbd: debug2: DbdBackupHost = (null) slurmdbd: debug2: DbdHost = hannibal-hn slurmdbd: debug2: DbdPort = 7032 slurmdbd: debug2: DebugFlags = (null) slurmdbd: debug2: DebugLevel = 6 slurmdbd: debug2: DebugLevelSyslog = 10 slurmdbd: debug2: DefaultQOS = (null) slurmdbd: debug2: LogFile = /var/log/slurmdbd.log slurmdbd: debug2: MessageTimeout = 100 slurmdbd: debug2: Parameters = (null) slurmdbd: debug2: PidFile = /run/slurmdbd.pid slurmdbd: debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm slurmdbd: debug2: PrivateData = none slurmdbd: debug2: PurgeEventAfter = 1 months* slurmdbd: debug2: PurgeJobAfter = 12 months* slurmdbd: debug2: PurgeResvAfter = 1 months* slurmdbd: debug2: PurgeStepAfter = 1 months slurmdbd: debug2: PurgeSuspendAfter = 1 months slurmdbd: debug2: PurgeTXNAfter = 12 months slurmdbd: debug2: PurgeUsageAfter = 24 months slurmdbd: debug2: SlurmUser = root(0) slurmdbd: debug2: StorageBackupHost = (null) slurmdbd: debug2: StorageHost = localhost slurmdbd: debug2: StorageLoc = slurm_acct_db slurmdbd: debug2: StoragePort = 3306 slurmdbd: debug2: StorageType = accounting_storage/mysql slurmdbd: debug2: StorageUser = slurm slurmdbd: debug2: TCPTimeout = 2 slurmdbd: debug2: TrackWCKey = 0 slurmdbd: debug2: TrackSlurmctldDown= 0 slurmdbd: debug2: acct_storage_p_get_connection: request new connection 1 slurmdbd: debug2: Attempting to connect to localhost:3306 slurmdbd: slurmdbd version 19.05.5 started slurmdbd: debug2: running rollup at Thu May 30 13:50:08 2024 slurmdbd: debug2: Everything rolled up

It goes like this for some time and then it crashes with this message

slurmdbd: Terminate signal (SIGINT or SIGTERM) received slurmdbd: debug: rpc_mgr shutting down

On Thu, May 30, 2024 at 8:18 AM mercan <ahmet.mercan@uhem.itu.edu.trmailto:ahmet.mercan@uhem.itu.edu.tr> wrote:

Did you try to connect database using mysql command?

mysql --user=slurm --password=slurmdbpass slurm_acct_db

C. Ahmet Mercan

On 30.05.2024 14:48, Radhouane Aniba via slurm-users wrote: Thank you Ahmet, I dont have a firewall active. And because slurmdbd cannot connect to the database I am not able to getting it to be activated through systemctl I will share the output for slurmdbd -D -vvv shortly but overall it is always saying trying to connect to the db and then retries a couple of times and crashes

R.

On Thu, May 30, 2024 at 2:51 AM mercan <ahmet.mercan@uhem.itu.edu.trmailto:ahmet.mercan@uhem.itu.edu.tr> wrote:

Hi;

Did you check can you connect db with your conf parameters from head-node:

mysql --user=slurm --password=slurmdbpass slurm_acct_db

Also, check and stop firewall and selinux, if they are running.

Last, you can stop slurmdbd, then run run terminal with:

slurmdbd -D -vvv

Regards;

C. Ahmet Mercan

On 30.05.2024 00:05, Radhouane Aniba via slurm-users wrote: Hi everyone I am trying to get slurmdbd to run on my local home server but I am really struggling. Note : am a novice slurm user my slurmdbd always times out even though all the details in the conf file are correct

My log looks like this

[2024-05-29T20:51:30.088] Accounting storage MYSQL plugin loaded [2024-05-29T20:51:30.088] debug2: ArchiveDir = /tmp [2024-05-29T20:51:30.088] debug2: ArchiveScript = (null) [2024-05-29T20:51:30.088] debug2: AuthAltTypes = (null) [2024-05-29T20:51:30.088] debug2: AuthInfo = (null) [2024-05-29T20:51:30.088] debug2: AuthType = auth/munge [2024-05-29T20:51:30.088] debug2: CommitDelay = 0 [2024-05-29T20:51:30.088] debug2: DbdAddr = localhost [2024-05-29T20:51:30.088] debug2: DbdBackupHost = (null) [2024-05-29T20:51:30.088] debug2: DbdHost = head-node [2024-05-29T20:51:30.088] debug2: DbdPort = 7032 [2024-05-29T20:51:30.088] debug2: DebugFlags = (null) [2024-05-29T20:51:30.088] debug2: DebugLevel = 6 [2024-05-29T20:51:30.088] debug2: DebugLevelSyslog = 10 [2024-05-29T20:51:30.088] debug2: DefaultQOS = (null) [2024-05-29T20:51:30.088] debug2: LogFile = /var/log/slurmdbd.log [2024-05-29T20:51:30.088] debug2: MessageTimeout = 100 [2024-05-29T20:51:30.088] debug2: Parameters = (null) [2024-05-29T20:51:30.088] debug2: PidFile = /run/slurmdbd.pid [2024-05-29T20:51:30.088] debug2: PluginDir = /usr/lib/x86_64-linux-gnu/slurm-wlm [2024-05-29T20:51:30.088] debug2: PrivateData = none [2024-05-29T20:51:30.088] debug2: PurgeEventAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeJobAfter = 12 months* [2024-05-29T20:51:30.088] debug2: PurgeResvAfter = 1 months* [2024-05-29T20:51:30.088] debug2: PurgeStepAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeSuspendAfter = 1 months [2024-05-29T20:51:30.088] debug2: PurgeTXNAfter = 12 months [2024-05-29T20:51:30.088] debug2: PurgeUsageAfter = 24 months [2024-05-29T20:51:30.088] debug2: SlurmUser = root(0) [2024-05-29T20:51:30.089] debug2: StorageBackupHost = (null) [2024-05-29T20:51:30.089] debug2: StorageHost = localhost [2024-05-29T20:51:30.089] debug2: StorageLoc = slurm_acct_db [2024-05-29T20:51:30.089] debug2: StoragePort = 3306 [2024-05-29T20:51:30.089] debug2: StorageType = accounting_storage/mysql [2024-05-29T20:51:30.089] debug2: StorageUser = slurm [2024-05-29T20:51:30.089] debug2: TCPTimeout = 2 [2024-05-29T20:51:30.089] debug2: TrackWCKey = 0 [2024-05-29T20:51:30.089] debug2: TrackSlurmctldDown= 0 [2024-05-29T20:51:30.089] debug2: acct_storage_p_get_connection: request new connection 1 [2024-05-29T20:51:30.089] debug2: Attempting to connect to localhost:3306 [2024-05-29T20:51:30.090] slurmdbd version 19.05.5 started [2024-05-29T20:51:30.090] debug2: running rollup at Wed May 29 20:51:30 2024 [2024-05-29T20:51:30.091] debug2: Everything rolled up [2024-05-29T20:51:49.673] Terminate signal (SIGINT or SIGTERM) received [2024-05-29T20:51:49.673] debug: rpc_mgr shutting down

my config file looks like this

ArchiveEvents=yes ArchiveJobs=yes ArchiveResvs=yes ArchiveSteps=no ArchiveSuspend=no ArchiveTXN=no ArchiveUsage=no PurgeEventAfter=1month PurgeJobAfter=12month PurgeResvAfter=1month PurgeStepAfter=1month PurgeSuspendAfter=1month PurgeTXNAfter=12month PurgeUsageAfter=24month # Authentication info AuthType=auth/munge # slurmDBD info DbdAddr=localhost DbdHost=head-node DbdPort=7032 SlurmUser=root MessageTimeout=100 DebugLevel=5 #DefaultQOS=normal,standby LogFile=/var/log/slurmdbd.log PidFile=/run/slurmdbd.pid #PrivateData=accounts,users,usage,jobs #TrackWCKey=yes # # Database info StorageType=accounting_storage/mysql StorageHost=localhost StoragePort=3306 StoragePass=slurmdbpass StorageUser=slurm StorageLoc=slurm_acct_db I used standard names and passwords to get started and I will change later

but everytime I try to start slurmdbd.service it crashes and I have that log that I shared with you

I use these versions

slurmdbd -V slurm-wlm 19.05.5 mysql Ver 15.1 Distrib 10.3.39-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 Everything else Is working properly except I cannot get slurmdbd to work and at this point I exhausted all my possible trials :) looking for some expert insights :)

Any idea what I am doing wrong here ? Also I didn't compile any slurm package. I used the binary from apt repos

Any help will be appreciated

Cheers

Rad

--

-- Rad Aniba, PhD

-- Benjamin Smith bsmith5@ed.ac.ukmailto:bsmith5@ed.ac.uk Computing Officer, AT-7.12a Research and Teaching Unit School of Informatics, University of Edinburgh

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.