0 0 Share PDF

Swarm join fails with transport error on 18.09 engine with http proxy configuration

Article ID: KB000935

Issue

If you have configured the Docker Engine 18.09 with the HTTP_PROXY or HTTPS_PROXY environment variables, then the following error may prevent joining or promoting manager nodes:

Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup proxy.example.com on 192.168.121.1:53: no such host"

This error can be found on the command line in response to docker swarm join with a manager token, or in docker info on the node that is being promoted after promotion fails.

An additional symptom of this issue is the UCP node error error: swarm mode manager has no manager status.

An example of a community issue report demonstrating this issue is moby/moby#36951.

Prerequisites

  • Docker Engine 18.09.0x prior to 18.09.04
  • Joining or promoting a manager node
  • HTTP HTTPS proxy configured (configuration visible docker info)

Root Cause

An update to the Docker Swarm gRPC libraries unintentionally caused them to start respecting proxy environment variables HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

Resolution

A patch at https://github.com/docker/swarmkit/pull/2802 was created to address the issue by restoring the original behavior of ignoring proxy environment variables. This patch was released in Docker Engine 18.09.4. You can find it in the EE Engine 18.04 Release Notes under entry:

Fixed issue for swarm nodes not being able to join as masters if http proxy is set.

Work Around

Add all manager IP addresses to the NO_PROXY environment variable using the instructions at docs.docker.com

Notes:

  • You must specify individual manager node Swarm IPs in NO_PROXY. Docker EE Engine 18.09.3 and lower do not support CIDR notation for NO_PROXY.
  • You must restart the docker systemd service in order for the change in environment variables to be picked up by the daemon.

What's Next