0 0 Share PDF

How to workaround a failed promote/demote (rethinkdb/etcd mismatch)

Article ID: KB000372

Workaround failed promote/demote (rethinkdb/etcd mismatch) within UCP

Under certain circumstances promotion or demotion may get stuck, resulting in a manager node reporting [Pending] Node is being reconfigured (https://success.docker.com/api/asset/.%2Fhow-to-workaround-a-failed-promote-demote-rethinkdb-etcd-mismatch%2F11%)


If you view the Agent Log the following failure information will be reported

[ERROR] Unable to promote node to controller: unsuccessful node promote request: node promotion failed: etcd and rethinkdb cluster healthcheck failed: mismatch between etcd (https://success.docker.com/api/asset/.%2Fhow-to-workaround-a-failed-promote-demote-rethinkdb-etcd-mismatch%2F2) and rethinkdb (1) replica count


Recovery Procedure

Determining the actual number of expected controllers in the system, then run the following command (replacing "2" with the actual number for your environment)

docker ps --filter name=ucp-kv -q | wc -l

docker exec ucp-auth-api enzi $(docker inspect --format '{{ index .Args 0 }}' ucp-auth-api) --debug reconfigure-db --num-replicas 2