0 0 Share PDF

How to increase the ARP cache garbage collection threshold

Article ID: KB000148

Issue

A typical service is attached to multiple networks. At a minimum, there is a user-defined overlay network, the docker_gwbridge overlay network for egress to the Internet, and the ingress overlay for IPVS ingress routing into the swarm. For N replicated tasks in a service, the service will also occupy N IP addresses on that service's overlay network. It's easy to see that with a large number of services, replicated numerous times, that the number of IP addresses needed may grow to be very large. If these containers/tasks are on the same overlay network, they must be programmed with the information of all their peers on the network for service discovery. It's possible, with very large networks, to exhaust the limits of the ARP cache on the host, leading to errors like this from dmesg:

neighbour: arp_cache: neighbor table overflow!

The ARP cache is a lookup table of IP/MAC address bindings. This problem may exhibit itself when that table grows large enough for the kernel to trigger ARP cache garbage collection.

There are sysctl tunables that may be set to increase the size of the ARP cache on Linux hosts.

Resolution

To increase the size of the ARP cache, there are 3 sysctl tunables that may be set:

  • sysctl -w net.ipv4.neigh.default.gc_thresh1=<n>: gc_thresh1 represents the minimum number of entries that may be in the ARP cache. Garbage collection will not be triggered if the number of entries is below this setting.

  • sysctl -w net.ipv4.neigh.default.gc_thresh2=<n>: gc_thresh2 represents the soft maximum number of entries that may be in the ARP cache. This setting is arguably the most important, as ARP garbage collection will be triggered ~5s after reaching this soft maximum.

  • sysctl -w net.ipv4.neigh.default.gc_thresh3=<n>: gc_thresh3 represents the hard maximum number of entries in the ARP cache.

Docker recommends estimating the maximum number of replicated services that you will operate in the swarm and setting the ARP cache sizes on your Docker hosts to an appropriate level.

Additional Resources