0 0 Share PDF

How to identify IP exhaustion in Swarm Mode overlay networks

Issue

Swarm Mode overlay networks default to /24 sized subnets, or approximately 256 IPs. Each service deployed may consume multiple IP addresses, as well as one each node peering on a given overlay network. Sometimes, in the course of daily operations, the total # of IP addresses may be exhausted on a given overlay network.

with Docker Engine 17.06

When looking at docker service ps --no-trunc service you may notice that there are some tasks stuck in the New state and never transition to Running. The engine does not surface an "ip exhaustion" error to the console during docker service ps.

with Docker Engine 18.09

You should see troubleshooting information from docker service ps --no-trunc service explaining that the overlay network has run out of available IPs. Specifically, an error message stating node is missing network attachments, ip addresses may be exhausted.

$ docker service ps --no-trunc vigorous_newton --format {{ .ID}}
ID                          NAME                    IMAGE                     NODE               DESIRED STATE       CURRENT STATE               ERROR
n9ujsy78uziwefwhar8acc94u   vigorous_newton.1       nginx:latest@sha256:...   ip-172-31-12-191   Running             Preparing 14 seconds ago
xm2yvsnazmx1oxjmwojnhidvy    \_ vigorous_newton.1   nginx:latest@sha256:...   ip-172-31-34-126   Shutdown            Rejected 19 seconds ago    "node is missing network attachments, ip addresses may be exhausted"

Docker Engine logs

With all versions of the Docker engine, there will be accompanying logs from the Docker daemon in syslog or journald corresponding to this error on the swarm mode leader at the time when the task was scheduled:

level=error msg="task allocation failure" error="failed to allocate network IP for task taskid network networkid : could not find an available IP" module=node
level=error msg="Failed to allocate network resources for node nodeid" error="could not find an available IP" module=node node.id=nodeid

Resolution

The most straightforward way to move past this problem is to scale down the number of services attached to this network and re-deploy them elsewhere. It's generally recommended to only deploy services that need to communicate with each other on the same network.

For certain use cases, however, it may be desirable to create a larger subnet. To expand the subnet on a network requires that the network be destroyed and recreated. It's not possible to re-size an existing overlay subnet. The best way to prevent an outage in this situation is to ensure that your services can gracefully re-connect to each other. Modifying the network while services are deployed will necessarily involve a change in IP address; therefore if your application is not able to gracefully initiate a new connection, it may need to to be restarted in order to re-initiate a connection. This may involve some downtime for your application, so plan accordingly.

It's generally recommended to plan for network sizes of /24. To create a custom subnet, you can specify a subnet in CIDR notation docker network create:

docker network create --driver overlay --subnet=192.168.0.0/16 large_net

You can modify an existing service's networks with the --network-rm and --network-add flags:

docker service update laughing_buddha --network-rm small_net --network-add large_net

What's Next