<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>It could be a problem with ARP cache.</p>
<p>If the number of devices approaches 512, there is a kernel
limitation in dynamic ARP-cache size and it can result in the loss
of connectivity between nodes. <br>
</p>
<p>The garbage collector will run if the number of entries in the
cache is less than 128, by default:</p>
<p><strong>gc_thresh1</strong><br>
The minimum number of entries to keep in the ARP cache. The
garbage collector will not run if there are fewer than this number
of entries in the cache. Defaults to 128. <br>
<strong>gc_thresh2</strong><br>
The soft maximum number of entries to keep in the ARP cache. The
garbage collector will allow the number of entries to exceed this
for 5 seconds before collection will be performed. Defaults to
512. <br>
<strong>gc_thresh3</strong><br>
The hard maximum number of entries to keep in the ARP cache. The
garbage collector will always run if there are more than this
number of entries in the cache. Defaults to 1024. <br>
</p>
<p>You can check: <br>
</p>
<pre>cat /proc/net/arp |wc</pre>
<p>If this is the case, increase the ARC cache garbe collection
parameters:</p>
<pre> #Force gc to clean-up quickly
sysctl -w net.ipv4.neigh.default.gc_interval = 3600
sysctl -w net.ipv6.neigh.default.gc_interval = 3600</pre>
<pre># Set ARP cache entry timeout
sysctl -w net.ipv4.neigh.default.gc_stale_time = 3600
sysctl -w net.ipv6.neigh.default.gc_stale_time = 3600
</pre>
<pre>#garbage collector</pre>
<pre>sysctl -w net.ipv4.neigh.default.gc_thresh1=2048
sysctl -w net.ipv4.neigh.default.gc_thresh2=4096
sysctl -w net.ipv4.neigh.default.gc_thresh3=8192</pre>
<pre>sysctl -w net.ipv6.neigh.default.gc_thresh1=2048
sysctl -w net.ipv6.neigh.default.gc_thresh2=4096
sysctl -w net.ipv6.neigh.default.gc_thresh3=8192</pre>
<p>Or just insert in /etc/sysctl.conf</p>
<p>Cheers,</p>
<p>Barbara<br>
</p>
<div class="moz-cite-prefix">On 5/15/19 5:02 AM, Bill Broadley
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:6bad7436-8724-9fb6-f296-85e314f32151@cse.ucdavis.edu">
<pre class="moz-quote-pre" wrap="">
My latest addition to a cluster results in a group of the same nodes periodically getting listed as
"not-responding" and usually (but not always) recovering.
I increased logging up to debug3 and see messages like:
[2019-05-14T17:09:25.247] debug: Spawning ping agent for
bigmem[1-9],bm[1,7,9-13],c6-[66-77,87-88,92-97],c8-[62-63],c9-[65,74-75],c10-[18,66-74,87-97],c11-[71-77,86-87,89-93,95-96]
And more problematic:
[2019-05-14T17:09:26.248] error: Nodes bm13,c6-[66-77,87-88,92-97] not responding
Out of 200 nodes, it's almost always those 20. Forward DNS (on both name servers), reverse DNS (on
both name servers), netmask, and /etc/hosts seem fine. It's possible the network hardware has a
problem, but some nodes on the same switch always work, some don't. Guess it could be an arp table
overflow or similar.
Despite a fair bit of testing, I've not been able to get any dns lookup or connection request
between the slurm controller and compute nodes, or compute to compute.
>From googling and searching the ticket system it seems like slurm builds a tree, then asks nodes to
check on the status of other nodes. From what I can tell if a node is listed as healthy, but can't
contact 20 other nodes, those 20 nodes are listed as not-responding. If that happens for longer
than the timer, then all those nodes go down.
So my questions:
1) can I run this "spawning ping agent" myself to help debug it?
2) can I get puppet to print out this tree so I can figure out which node
can not contact the nodes are being listed as down?
3) is there any other way to track down which node tried and failed to
contact the not-reponding
</pre>
</blockquote>
</body>
</html>