A typical service is attached to multiple networks. At a minimum, there is a user-defined overlay network, the
docker_gwbridge overlay network for egress to the Internet, and the
ingress overlay for IPVS ingress routing into the swarm. For N replicated tasks in a service, the service will also occupy N IP addresses on that service's overlay network. It's easy to see that with a large number of services, replicated numerous times, that the number of IP addresses needed may grow to be very large. If these containers/tasks are on the same overlay network, they must be programmed with the information of all their peers on the network for service discovery. It's possible, with very large networks, to exhaust the limits of the ARP cache on the host, leading to errors like this from
neighbour: arp_cache: neighbor table overflow!
The ARP cache is a lookup table of IP/MAC address bindings. This problem may exhibit itself when that table grows large enough for the kernel to trigger ARP cache garbage collection.
There are sysctl tunables that may be set to increase the size of the ARP cache on Linux hosts.
To increase the size of the ARP cache, there are 3 sysctl tunables that may be set:
sysctl -w net.ipv4.neigh.default.gc_thresh1=<n>:
gc_thresh1represents the minimum number of entries that may be in the ARP cache. Garbage collection will not be triggered if the number of entries is below this setting.
sysctl -w net.ipv4.neigh.default.gc_thresh2=<n>:
gc_thresh2represents the soft maximum number of entries that may be in the ARP cache. This setting is arguably the most important, as ARP garbage collection will be triggered ~5s after reaching this soft maximum.
sysctl -w net.ipv4.neigh.default.gc_thresh3=<n>:
gc_thresh3represents the hard maximum number of entries in the ARP cache.
Docker recommends estimating the maximum number of replicated services that you will operate in the swarm and setting the ARP cache sizes on your Docker hosts to an appropriate level.