Skip to main content

Docker Success Center

The Docker enterprise customer portal.

Docker, Inc.

How to workaround a failed promote/demote (rethinkdb/etcd mismatch)

Workaround failed promote/demote (rethinkdb/etcd mismatch)

Under certain circumstances promotion or demotion may get stuck, resulting in a manager node reporting [Pending] Node is being reconfigured (11%)

Details

If you view the Agent Log the following failure information will be reported

[ERROR] Unable to promote node to controller: unsuccessful node promote request: node promotion failed: etcd and rethinkdb cluster healthcheck failed: mismatch between etcd (2) and rethinkdb (1) replica count

Log

Recovery Procedure

Determining the actual number of expected controllers in the system, then run the following command (replacing "2" with the actual number for your environment)

docker ps --filter name=ucp-kv -q | wc -l
docker exec ucp-auth-api enzi $(docker inspect --format '{{ index .Args 0 }}' ucp-auth-api) --debug reconfigure-db --num-replicas 2