<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Apologies for not having more concrete information available when I’m replying to you, but I figured maybe having a fast hint might be better.
<div><br>
</div>
<div>Have a look at how the various daemons communicate with one another. This sounds to me like a firewall thing between maybe the SlurmCtld and where the SlurmDBD is running right now, or vice-versa or something like that. The “scontrol show cluster” thing
is a giveaway. That is populated dynamically, not pulled from a config file exactly. </div>
<div><br>
</div>
<div>I ran into this exact thing years ago, but can’t remember where the firewall was the issue. <br>
<br>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jun 15, 2022, at 20:12, Reed Dier <reed.dier@focusvq.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr"> Hoping this is an easy answer.
<div class=""><br class="">
</div>
<div class="">My mysql instance somehow corrupted itself, and I’m having to purge and start over.</div>
<div class="">This is ok, because the data in there isn’t too valuable, and we aren’t making use of associations or anything like that yet (no AccountingStorageEnforce).</div>
<div class=""><br class="">
</div>
<div class="">That said, I’ve decided to put the dbd’s mysql instance on my main database server, rather than in a small vm alongside the dbd.</div>
<div class="">Jobs are still submitting alright, and after adding the cluster back with `sacctmgr create cluster $cluster` it seems to have stopped the log firehose.</div>
<div class="">The issue I’m mainly seeing now is in the dbd logs:</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:40:43.064] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
<div class=""><font face="Menlo" class="">[2022-06-15T19:40:43.065] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:45:39.827] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:48:01.038] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:48:01.039] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:48:38.104] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:50:39.290] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">[2022-06-15T19:55:39.769] error: _add_registered_cluster: trying to register a cluster ($cluster) with no remote port</font></div>
</blockquote>
<div class=""><br class="">
</div>
And if I run </div>
<div class="">
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">$ sacctmgr show cluster</font></div>
<div class=""><font face="Menlo" class=""> Cluster ControlHost ControlPort RPC Share GrpJobs GrpTRES GrpSubmit MaxJobs MaxTRES MaxSubmit MaxWall QOS Def QOS</font></div>
<div class=""><font face="Menlo" class=""> ---------- --------------- ------------ ----- --------- ------- ------------- --------- ------- ------------- --------- ----------- -------------------- ---------</font></div>
<div class=""><font face="Menlo" class=""> $cluster 0 0 1 normal</font></div>
</blockquote>
<br class="">
I can see the ControlHost, ControlPort, and RPC are all missing.</div>
<div class="">So I’m not sure what I need to do to figure out how to effectively reset my dbd.</div>
<div class="">Also, $cluster in sacctmgr matches ClusterName=$cluster in my slurm.conf.</div>
<div class=""><br class="">
</div>
<div class="">The only thing that has changed is the StorageHost in the dbd conf, and I made the database, user, and grant all on slurm_acct_db.*, on the new database server.</div>
<div class="">And I’ve verified that it has made tables, and that I can connect from the host with the correct credentials.</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">
<div class=""><font face="Menlo" class="">mysql> show tables;</font></div>
<div class=""><font face="Menlo" class="">+----------------------------------+</font></div>
<div class=""><font face="Menlo" class="">| Tables_in_slurm_acct_db |</font></div>
<div class=""><font face="Menlo" class="">+----------------------------------+</font></div>
<div class=""><font face="Menlo" class="">| acct_coord_table |</font></div>
<div class=""><font face="Menlo" class="">| acct_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_assoc_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_assoc_usage_day_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_assoc_usage_hour_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_assoc_usage_month_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_event_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_job_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_last_ran_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_resv_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_step_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_suspend_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_usage_day_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_usage_hour_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_usage_month_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_wckey_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_wckey_usage_day_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_wckey_usage_hour_table |</font></div>
<div class=""><font face="Menlo" class="">| $cluster_wckey_usage_month_table |</font></div>
<div class=""><font face="Menlo" class="">| clus_res_table |</font></div>
<div class=""><font face="Menlo" class="">| cluster_table |</font></div>
<div class=""><font face="Menlo" class="">| convert_version_table |</font></div>
<div class=""><font face="Menlo" class="">| federation_table |</font></div>
<div class=""><font face="Menlo" class="">| qos_table |</font></div>
<div class=""><font face="Menlo" class="">| res_table |</font></div>
<div class=""><font face="Menlo" class="">| table_defs_table |</font></div>
<div class=""><font face="Menlo" class="">| tres_table |</font></div>
<div class=""><font face="Menlo" class="">| txn_table |</font></div>
<div class=""><font face="Menlo" class="">| user_table |</font></div>
<div class=""><font face="Menlo" class="">+----------------------------------+</font></div>
<div class=""><font face="Menlo" class="">29 rows in set (0.01 sec)</font></div>
</blockquote>
</div>
<div class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class="">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Any tips are appreciated.</div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">21.08.7 and Ubuntu 20.04.</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">Slurmdbd and slurmctld(1) are running on one host, and slurmctld(2) is running on another host, and is the primary.</div>
</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Reed</div>
</div>
</blockquote>
</div>
</body>
</html>