<div dir="ltr"><div>Please do three things for the list:</div><div><br></div><div>a) cat /etc/*elease* <br></div><div><br></div><div>b) give details on how Slurm was installed on the master node and the compute nodes</div><div><br></div><div>c) How was your slurm.conf file created? Is this file identical on master node and compute nodes?</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 15 June 2018 at 11:26, UGI <span dir="ltr"><<a href="mailto:ugiwgh@gmail.com" target="_blank">ugiwgh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I didn't have the directory /var/spool/slurmctld/. And then I mkdir the directory, and "chown slurm:slurm /var/spool/slurmctld". <div>But there is also the errors.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-06-15 16:00 GMT+08:00 John Hearns <span dir="ltr"><<a href="mailto:hearnsj@googlemail.com" target="_blank">hearnsj@googlemail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And your permissions on the directory /var/spool/slurmctld/ are ????<br></div><div class="m_-8809956576684573042HOEnZb"><div class="m_-8809956576684573042h5"><div class="gmail_extra"><br><div class="gmail_quote">On 15 June 2018 at 09:11, UGI <span dir="ltr"><<a href="mailto:ugiwgh@gmail.com" target="_blank">ugiwgh@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">When I start slurmctld, there are some errors in log. And the job running information doesn't store to mysql via slurmdbd.<div><br></div><div>I set </div><div>
<span></span>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">AccountingStoragePass=/usr/loc<wbr>al/munge-munge-0.5.13/var/run/<wbr>munge/munge.socket.2</span></p><p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span style="font-variant-ligatures:no-common-ligatures">AccountingStorageType=accounti<wbr>ng_storage/slurmdbd</span></p><p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span style="font-variant-ligatures:no-common-ligatures">
<span></span>
</span></p><p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;line-height:normal;background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">JobAcctGatherType=jobacct_gath<wbr>er/linux</span></p><p></p><div>in slurm.conf.</div><div><br></div><div><br><div>The following message is the log which slurmctld output.</div><div>
<span></span>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:05:44.763] Terminate signal (SIGINT or SIGTERM) received</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:05:44.807] Saving all slurm state</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:05:45.101] error: slurmdbd: Sending fini msg: No error</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:05:45.126] layouts: all layouts are now unloaded.</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.761] slurmctld version 17.11.7 started on cluster myslurm</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.785] error: slurm_persist_conn_open_withou<wbr>t_init: failed to open persistent connection to localhost:6819: Connection refused</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.785] error: slurmdbd: Sending PersistInit msg: Connection refused</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.785] error: slurmdbd: Sending PersistInit msg: Connection refused</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.787] layouts: no layout to initialize</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] error: ##############################<wbr>##################</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] error: ### <span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-Apple-converted-space"> </span>SEVERE SECURITY VULERABILTY<span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-Apple-converted-space"> </span>###</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] error: ### StateSaveLocation DIRECTORY IS WORLD WRITABLE ###</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] error: ### <span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-Apple-converted-space"> </span>CORRECT FILE PERMISSIONS <span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-Apple-converted-space"> </span>###</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] error: ##############################<wbr>##################</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] layouts: loading entities/relations information</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] Recovered state of 1 nodes</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.824] Recovered JobID=12 State=0x3 NodeCnt=0 Assoc=2</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] Recovered information about 1 jobs</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] cons_res: select_p_node_init</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] cons_res: preparing for 1 partitions</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] Recovered state of 0 reservations</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] _preserve_plugins: backup_controller not specified</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] cons_res: select_p_reconfigure</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] cons_res: select_p_node_init</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] cons_res: preparing for 1 partitions</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] Running as primary controller</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] Registering slurmctld at port 6817 with slurmdbd.</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] error: slurmdbd: Sending PersistInit msg: Connection refused</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.825] error: slurmdbd: Sending PersistInit msg: Connection refused</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.826] No parameter for mcs plugin, default values set</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:07.826] mcs: MCSParameters = (null). ondemand set.</span></p>
<p class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-weight:normal;font-stretch:normal;font-size:12px;line-height:normal;font-family:"Andale Mono";color:rgb(40,254,20);background-color:rgba(0,0,0,0.9)"><span class="m_-8809956576684573042m_-1389347955944861780m_-2936897505767954121gmail-s1" style="font-variant-ligatures:no-common-ligatures">[2018-06-15T11:06:10.829] SchedulerParameters=default_qu<wbr>eue_depth=100,max_rpc_cnt=0,ma<wbr>x_sched_time=2,partition_job_d<wbr>epth=0,sched_max_job_start=0,s<wbr>ched_min_interval=2</span></p>
<br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>