<html><head><meta http-equiv="content-type" content="text/html; charset=GB2312"><style>body { line-height: 1.5; }body { font-size: 14px; font-family: 'Microsoft YaHei UI'; color: rgb(0, 0, 0); line-height: 1.5; }</style></head><body>
<div><span></span>Dear there,</div>
<div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> We tested mpi allreduce job in three modes (srun-dtcp 、mpirun-slurm、mpirun-ssh), and we found that the job running time in the mpirun-ssh mode is shorter than the other modes. </span></div><div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> We've set parameters like below:</span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span>/usr/lib/systemd/system/slurmd.service:</span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="line-height: 1.5;">LimitMEMLOCK=infinity</span></span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="line-height: 1.5;">LimitSTACK=infinity</span></span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span>/etc/sysconfig/slurmd:</span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="line-height: 1.5;">ulimit -l unlimited</span></span></div><div><span style="background-color: rgb(255, 255, 153);"><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="color: rgb(0, 0, 0); line-height: 1.5;"> </span><span style="line-height: 1.5;">ulimit -s unlimited</span></span></div><div><span style="color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> </span><span style="line-height: 1.5; background-color: transparent;">We want to know if this is normal ?</span><span style="line-height: 1.5; orphans: 2; white-space: pre-wrap; widows: 2; background-color: transparent;">Will functions such as cgroup and pam_slurm_adopt limit job performance ? And </span><span style="line-height: 1.5; background-color: transparent;">how to improve the efficiency of slurm jobs.</span></div></div><div><span style="color: rgb(0, 0, 0); line-height: 1.5; background-color: rgba(0, 0, 0, 0);"> Here is the test results:</span></div><!--StartFragment--><table style="border-collapse:collapse" selecttype="cells"><colgroup><col width="123" style="width: 123px;"><col width="86" style="width: 86px;"><col width="102.23" style="width: 102.23px;"><col width="93.76" style="width: 93.76px;"></colgroup><tbody><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-weight:bold;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">size</td><td style="font-size:11pt;font-weight:bold;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">srun-dtcp</td><td style="font-size:11pt;font-weight:bold;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">mpirun-slurm</td><td style="font-size:11pt;font-weight:bold;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">mpirun-ssh</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">0</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">0.05</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">0.06</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">0.05</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">4</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">2551.83</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">355.67</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">281.02</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">8</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">1469.32</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">2419.97</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">139</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">16</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">67.41</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">151.87</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">1000.15</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">32</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">73.31</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">143.15</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">126.22</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">64</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">107.14</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">111.6</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">126.3</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">128</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">77.1</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">2473.62</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">344.36</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">256</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">46.39</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">417.95</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">65.09</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">512</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">92.84</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">260.69</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">90.5</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">1024</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">97.9</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">137.13</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">112.3</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">2048</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">286.27</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">233.21</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">169.59</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">4096</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">155.69</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">343.59</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">160.9</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">8192</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">261.02</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">465.78</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">151.29</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">16384</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">12518.04</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">13363.71</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">3869.58</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">32768</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">22071.6</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">11398.21</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">4842.32</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">65536</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">6041.2</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">6666.95</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">3368.58</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">131072</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">10436.11</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">18071.53</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">10909.76</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">262144</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">13802.22</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">24728.78</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">8263.53</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">524288</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">13086.26</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">16394.72</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">4825.51</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">1048576</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">28193.08</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">15943.26</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">6490.29</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">2097152</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">63277.73</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">24411.58</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined="">15361.7</td></tr><tr height="24" style="mso-height-source:auto;height:24px;"><td style="font-size:11pt;font-family:等线;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;vertical-align:bottom;text-align:right;mso-number-format: 'General';" undefined="">4194304</td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined=""><span style="background-color: rgb(255, 255, 153);">58538.05</span></td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined=""><span style="background-color: rgb(255, 255, 153);">60516.15</span></td><td style="font-size:10pt;border-top: .5pt solid #000000;border-right: .5pt solid #000000;border-bottom: .5pt solid #000000;border-left: .5pt solid #000000;text-align:right;" undefined=""><span style="background-color: rgb(255, 255, 153);">33955.49</span></td></tr></tbody></table><br><b>(1)srun-dtcp job:</b><div><span microsoft="" yahei="" ui';="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">#!/bin/bash<br>#SBATCH -J test<br>#SBATCH -N 32<br>#SBATCH --ntasks-per-node=30<br>#SBATCH -p seperate<br><br>NP=$SLURM_NTASKS<br>srun --mpi=pmix_v3 /public/software/benchmark/imb/hpcx/2017/IMB-MPI1 -npmin $NP Allreduce</span></div><div><br></div><div><span microsoft="" yahei="" ui';="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><b>(2)mpirun-slurm job:</b></span></div><div><span microsoft="" yahei="" ui';="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'=""><span microsoft="" yahei="" ui';="" font-size:="" 14px;="" color:="" rgb(0,="" 0,="" 0);="" background-color:="" rgba(0,="" font-weight:="" normal;="" font-style:="" normal;text-decoration:="" none;'="">#!/bin/bash<br>#SBATCH -J test<br>#SBATCH -N 32<br>#SBATCH --ntasks-per-node=30<br>#SBATCH -p seperate<br><br>NP=$SLURM_NTASKS<br>mpirun /public/software/benchmark/imb/hpcx/2017/IMB-MPI1 -npmin $NP Allreduce</span></span></div><div><br></div><div><b>(3)<span style="line-height: 1.5; background-color: transparent;">mpirun-ssh job:</span></b></div><div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">#!/bin/bash</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">#SBATCH -J test</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">#SBATCH -N 32</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">#SBATCH --ntasks-per-node=30</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">#SBATCH -p seperate</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;"><br></div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">env | grep SLURM > env.log </div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">scontrol show hostname > nd.$SLURM_JOBID</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">NNODE=$SLURM_NNODES</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">NP=$SLURM_NTASKS</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;"><br></div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;">mpirun -np $NP -machinefile nd.$SLURM_JOBID -map-by ppr:30:node \</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;"> -mca plm rsh -mca plm_rsh_no_tree_spawn 1 -mca plm_rsh_num_concurrent $NNODE \</div><div yne-bulb-block="paragraph" style="white-space: pre-wrap; line-height: 1.75;"> /public/software/benchmark/imb/hpcx/2017/IMB-MPI1 -npmin $NP Allreduce</div><!--5f39ae17-8c62-4a45-bc43-b32064c9388a:W3siYmxvY2tJZCI6Ijg2ODItMTYwODYyMTEwNTI0NCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImFsaWduIjoibGVmdCIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiIjIn0seyJjaGFyIjoiISJ9LHsiY2hhciI6Ii8ifSx7ImNoYXIiOiJiIn0seyJjaGFyIjoiaSJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiIvIn0seyJjaGFyIjoiYiJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoiaCJ9XSwiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZX19LHsiYmxvY2tJZCI6IjQzODMtMTYwODc3MjYyNjk1OCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiIjIn0seyJjaGFyIjoiUyJ9LHsiY2hhciI6IkIifSx7ImNoYXIiOiJBIn0seyJjaGFyIjoiVCJ9LHsiY2hhciI6IkMifSx7ImNoYXIiOiJIIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiJKIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6InQifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI5OTY3LTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbeyJjaGFyIjoiIyJ9LHsiY2hhciI6IlMifSx7ImNoYXIiOiJCIn0seyJjaGFyIjoiQSJ9LHsiY2hhciI6IlQifSx7ImNoYXIiOiJDIn0seyJjaGFyIjoiSCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiItIn0seyJjaGFyIjoiTiJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIzIn0seyJjaGFyIjoiMiJ9XSwiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZX19LHsiYmxvY2tJZCI6Ijc2OTctMTYwODc3MjYyNjk1OCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiIjIn0seyJjaGFyIjoiUyJ9LHsiY2hhciI6IkIifSx7ImNoYXIiOiJBIn0seyJjaGFyIjoiVCJ9LHsiY2hhciI6IkMifSx7ImNoYXIiOiJIIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiItIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6ImsifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoiLSJ9LHsiY2hhciI6InAifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoibyJ9LHsiY2hhciI6ImQifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiPSJ9LHsiY2hhciI6IjMifSx7ImNoYXIiOiIwIn1dLCJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlfX0seyJibG9ja0lkIjoiNTczMy0xNjA4NzcyNjI2OTU4IiwiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImJhY2stY29sb3IiOiIiLCJwYWRkaW5nIjoiIiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJ0eXBlIjoicGFyYWdyYXBoIiwicmljaFRleHQiOnsiZGF0YSI6W3siY2hhciI6IiMifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoiQiJ9LHsiY2hhciI6IkEifSx7ImNoYXIiOiJUIn0seyJjaGFyIjoiQyJ9LHsiY2hhciI6IkgifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiLSJ9LHsiY2hhciI6InAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJwIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoidCJ9LHsiY2hhciI6ImUifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI2NTI3LTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbXSwiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZX19LHsiYmxvY2tJZCI6IjkyODItMTYwODc3MjYyNjk1OCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiJlIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6InYifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoifCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiJnIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJwIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IlMifSx7ImNoYXIiOiJMIn0seyJjaGFyIjoiVSJ9LHsiY2hhciI6IlIifSx7ImNoYXIiOiJNIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ij4ifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJ2In0seyJjaGFyIjoiLiJ9LHsiY2hhciI6ImwifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoiZyJ9LHsiY2hhciI6IiAifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI4NDgzLTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbeyJjaGFyIjoicyJ9LHsiY2hhciI6ImMifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6InQifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoibyJ9LHsiY2hhciI6ImwifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6ImgifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoidyJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiJoIn0seyJjaGFyIjoibyJ9LHsiY2hhciI6InMifSx7ImNoYXIiOiJ0In0seyJjaGFyIjoibiJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiI+In0seyJjaGFyIjoiICJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJkIn0seyJjaGFyIjoiLiJ9LHsiY2hhciI6IiQifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoiTCJ9LHsiY2hhciI6IlUifSx7ImNoYXIiOiJSIn0seyJjaGFyIjoiTSJ9LHsiY2hhciI6Il8ifSx7ImNoYXIiOiJKIn0seyJjaGFyIjoiTyJ9LHsiY2hhciI6IkIifSx7ImNoYXIiOiJJIn0seyJjaGFyIjoiRCJ9XSwiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZX19LHsiYmxvY2tJZCI6IjU3NzYtMTYwODc3MjYyNjk1OCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiJOIn0seyJjaGFyIjoiTiJ9LHsiY2hhciI6Ik8ifSx7ImNoYXIiOiJEIn0seyJjaGFyIjoiRSJ9LHsiY2hhciI6Ij0ifSx7ImNoYXIiOiIkIn0seyJjaGFyIjoiUyJ9LHsiY2hhciI6IkwifSx7ImNoYXIiOiJVIn0seyJjaGFyIjoiUiJ9LHsiY2hhciI6Ik0ifSx7ImNoYXIiOiJfIn0seyJjaGFyIjoiTiJ9LHsiY2hhciI6Ik4ifSx7ImNoYXIiOiJPIn0seyJjaGFyIjoiRCJ9LHsiY2hhciI6IkUifSx7ImNoYXIiOiJTIn1dLCJpc1JpY2hUZXh0Ijp0cnVlLCJrZWVwTGluZUJyZWFrIjp0cnVlfX0seyJibG9ja0lkIjoiNjc2My0xNjA4NzcyNjI2OTU4IiwiYmxvY2tUeXBlIjoicGFyYWdyYXBoIiwic3R5bGVzIjp7ImFsaWduIjoibGVmdCIsImJhY2stY29sb3IiOiIiLCJwYWRkaW5nIjoiIiwiaW5kZW50IjowLCJ0ZXh0LWluZGVudCI6MCwibGluZS1oZWlnaHQiOjEuNzV9LCJ0eXBlIjoicGFyYWdyYXBoIiwicmljaFRleHQiOnsiZGF0YSI6W3siY2hhciI6Ik4ifSx7ImNoYXIiOiJQIn0seyJjaGFyIjoiPSJ9LHsiY2hhciI6IiQifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoiTCJ9LHsiY2hhciI6IlUifSx7ImNoYXIiOiJSIn0seyJjaGFyIjoiTSJ9LHsiY2hhciI6Il8ifSx7ImNoYXIiOiJOIn0seyJjaGFyIjoiVCJ9LHsiY2hhciI6IkEifSx7ImNoYXIiOiJTIn0seyJjaGFyIjoiSyJ9LHsiY2hhciI6IlMifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI2MDQwLTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbXSwiaXNSaWNoVGV4dCI6dHJ1ZSwia2VlcExpbmVCcmVhayI6dHJ1ZX19LHsiYmxvY2tJZCI6Ijc0MzUtMTYwODc3MjY2NzM0OCIsImJsb2NrVHlwZSI6InBhcmFncmFwaCIsInN0eWxlcyI6eyJhbGlnbiI6ImxlZnQiLCJiYWNrLWNvbG9yIjoiIiwicGFkZGluZyI6IiIsImluZGVudCI6MCwidGV4dC1pbmRlbnQiOjAsImxpbmUtaGVpZ2h0IjoxLjc1fSwidHlwZSI6InBhcmFncmFwaCIsInJpY2hUZXh0Ijp7ImRhdGEiOlt7ImNoYXIiOiJtIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6ImkifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoidSJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiLSJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJwIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiQifSx7ImNoYXIiOiJOIn0seyJjaGFyIjoiUCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiItIn0seyJjaGFyIjoibSJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJjIn0seyJjaGFyIjoiaCJ9LHsiY2hhciI6ImkifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6ImYifSx7ImNoYXIiOiJpIn0seyJjaGFyIjoibCJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6ImQifSx7ImNoYXIiOiIuIn0seyJjaGFyIjoiJCJ9LHsiY2hhciI6IlMifSx7ImNoYXIiOiJMIn0seyJjaGFyIjoiVSJ9LHsiY2hhciI6IlIifSx7ImNoYXIiOiJNIn0seyJjaGFyIjoiXyJ9LHsiY2hhciI6IkoifSx7ImNoYXIiOiJPIn0seyJjaGFyIjoiQiJ9LHsiY2hhciI6IkkifSx7ImNoYXIiOiJEIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiYSJ9LHsiY2hhciI6InAifSx7ImNoYXIiOiItIn0seyJjaGFyIjoiYiJ9LHsiY2hhciI6InkifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6InAifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiOiJ9LHsiY2hhciI6IjMifSx7ImNoYXIiOiIwIn0seyJjaGFyIjoiOiJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJvIn0seyJjaGFyIjoiZCJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiXFwifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI0MDM3LTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbeyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6ImwifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoiaCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiItIn0seyJjaGFyIjoibSJ9LHsiY2hhciI6ImMifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6InAifSx7ImNoYXIiOiJsIn0seyJjaGFyIjoibSJ9LHsiY2hhciI6Il8ifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6ImgifSx7ImNoYXIiOiJfIn0seyJjaGFyIjoibiJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJfIn0seyJjaGFyIjoidCJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJlIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6Il8ifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiJ3In0seyJjaGFyIjoibiJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIxIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii0ifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6ImEifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6ImwifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiXyJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJzIn0seyJjaGFyIjoiaCJ9LHsiY2hhciI6Il8ifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoidSJ9LHsiY2hhciI6Im0ifSx7ImNoYXIiOiJfIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6InUifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoidCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIkIn0seyJjaGFyIjoiTiJ9LHsiY2hhciI6Ik4ifSx7ImNoYXIiOiJPIn0seyJjaGFyIjoiRCJ9LHsiY2hhciI6IkUifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiXFwifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fSx7ImJsb2NrSWQiOiI2MzY3LTE2MDg3NzI2MjY5NTgiLCJibG9ja1R5cGUiOiJwYXJhZ3JhcGgiLCJzdHlsZXMiOnsiYWxpZ24iOiJsZWZ0IiwiYmFjay1jb2xvciI6IiIsInBhZGRpbmciOiIiLCJpbmRlbnQiOjAsInRleHQtaW5kZW50IjowLCJsaW5lLWhlaWdodCI6MS43NX0sInR5cGUiOiJwYXJhZ3JhcGgiLCJyaWNoVGV4dCI6eyJkYXRhIjpbeyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6Ii8ifSx7ImNoYXIiOiJwIn0seyJjaGFyIjoidSJ9LHsiY2hhciI6ImIifSx7ImNoYXIiOiJsIn0seyJjaGFyIjoiaSJ9LHsiY2hhciI6ImMifSx7ImNoYXIiOiIvIn0seyJjaGFyIjoicyJ9LHsiY2hhciI6Im8ifSx7ImNoYXIiOiJmIn0seyJjaGFyIjoidCJ9LHsiY2hhciI6IncifSx7ImNoYXIiOiJhIn0seyJjaGFyIjoiciJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiIvIn0seyJjaGFyIjoiYiJ9LHsiY2hhciI6ImUifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoiYyJ9LHsiY2hhciI6ImgifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiYSJ9LHsiY2hhciI6InIifSx7ImNoYXIiOiJrIn0seyJjaGFyIjoiLyJ9LHsiY2hhciI6ImkifSx7ImNoYXIiOiJtIn0seyJjaGFyIjoiYiJ9LHsiY2hhciI6Ii8ifSx7ImNoYXIiOiJoIn0seyJjaGFyIjoicCJ9LHsiY2hhciI6ImMifSx7ImNoYXIiOiJ4In0seyJjaGFyIjoiLyJ9LHsiY2hhciI6IjIifSx7ImNoYXIiOiIwIn0seyJjaGFyIjoiMSJ9LHsiY2hhciI6IjcifSx7ImNoYXIiOiIvIn0seyJjaGFyIjoiSSJ9LHsiY2hhciI6Ik0ifSx7ImNoYXIiOiJCIn0seyJjaGFyIjoiLSJ9LHsiY2hhciI6Ik0ifSx7ImNoYXIiOiJQIn0seyJjaGFyIjoiSSJ9LHsiY2hhciI6IjEifSx7ImNoYXIiOiIgIn0seyJjaGFyIjoiLSJ9LHsiY2hhciI6Im4ifSx7ImNoYXIiOiJwIn0seyJjaGFyIjoibSJ9LHsiY2hhciI6ImkifSx7ImNoYXIiOiJuIn0seyJjaGFyIjoiICJ9LHsiY2hhciI6IiQifSx7ImNoYXIiOiJOIn0seyJjaGFyIjoiUCJ9LHsiY2hhciI6IiAifSx7ImNoYXIiOiJBIn0seyJjaGFyIjoibCJ9LHsiY2hhciI6ImwifSx7ImNoYXIiOiJyIn0seyJjaGFyIjoiZSJ9LHsiY2hhciI6ImQifSx7ImNoYXIiOiJ1In0seyJjaGFyIjoiYyJ9LHsiY2hhciI6ImUifV0sImlzUmljaFRleHQiOnRydWUsImtlZXBMaW5lQnJlYWsiOnRydWV9fV0=--></div><div><br><!--EndFragment--><hr style="width: 210px; height: 1px;" color="#b5c4df" size="1" align="left">
<div><span><div style="MARGIN: 10px; FONT-FAMILY: verdana; FONT-SIZE: 10pt"><table width="100%" style="orphans: 2; widows: 2; font-family: 'Microsoft YaHei UI';"><tbody><tr><td width="100%"><font color="#2f2f2f" face="宋体"><span style="font-size: 15px;">Best wishes!<br>menglong</span></font></td></tr><tr><td><span style="font-size: 15px;"><font color="#2f2f2f" face="宋体"><br></font></span><img src="cid:_Foxmail.1@00a385e0-c6ad-e25d-07f0-04a26500f020" border="0"><span style="font-size: 15px;"><font color="#2f2f2f" face="宋体"><br><br></font></span></td></tr></tbody></table></div></span></div>
</div></body></html>