0 0 Share PDF

How to rotate your UCP cluster root CA

Article ID: KB000623

Issue

This article explains how to rotate your UCP cluster root CA and your UCP client root CA.

Rotate UCP Cluster Root CA

  1. First, set your UCP version and UCP agent image with the version of UCP you are running:

    export UCP_VERSION='2.1.6'
    export UCP_AGENT_IMG="docker/ucp-agent:$UCP_VERSION"
    
  2. Get the existing root CA certificate contents:

    export OLD_ROOT_CERT=$(docker run --rm -v ucp-cluster-root-ca:/pki --entrypoint cat $UCP_AGENT_IMG /pki/cert.pem)
    
  3. Generate a new Root CA Key:

    openssl ecparam -name prime256v1 -genkey -noout -out new_root_key.pem
    export NEW_ROOT_KEY=$(cat new_root_key.pem)
    
  4. Generate a new Root CA Cert (Enter . for most of the options (country name, organization, etc.) but use new-swarm-ca for the Common Name):

    openssl req -new -x509 -key new_root_key.pem  -out new_root_cert.pem -days 3650
    export NEW_ROOT_CERT=$(cat new_root_cert.pem)
    
  5. Pause reconciliation by running this command on all nodes:

    docker container create --name ucp-agent-pause docker/ucp-agent:2.1.6
    
  6. Rotate the Swarm CA to the New Root CA by running the following command against one of the manager nodes:

    docker swarm ca --rotate --ca-cert new_root_cert.pem --ca-key new_root_key.pem
    
  7. Run these commands against each Docker manager node to reconfigure the UCP manager components to trust either the old or new cluster CA:

    export ALL_CERT_VOLS='
        ucp-auth-api-certs
        ucp-auth-store-certs
        ucp-auth-worker-certs
        ucp-controller-client-certs
        ucp-kv-certs
        ucp-node-certs
    '
    for VOL in $ALL_CERT_VOLS; do
        docker run -i --rm -v $VOL:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/ca.pem' << EOF
        $OLD_ROOT_CERT
        $NEW_ROOT_CERT
        EOF
    done
    docker run -i --rm -v ucp-cluster-root-ca:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/cert.pem' << EOF
        $NEW_ROOT_CERT
    EOF
    docker run -i --rm -v ucp-cluster-root-ca:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/key.pem' << EOF
        $NEW_ROOT_KEY
    EOF
    export UCP_CONTAINERS='
        ucp-controller
        ucp-metrics
        ucp-swarm-manager
        ucp-auth-api
        ucp-auth-worker
        ucp-auth-store
        ucp-cluster-root-ca
        ucp-client-root-ca
        ucp-proxy
        ucp-kv
    '
    docker restart $UCP_CONTAINERS
    
  8. Next, ensure that all the managers still appear healthy. Then run these commands on each manager to force regeneration of most certificates:

    docker run --rm -v ucp-controller-client-certs:/pki --entrypoint rm $UCP_AGENT_IMG /pki/cert.pem
    docker rm -f ucp-controller
    docker rm ucp-agent-pause
    
  9. Reconciliation is expected to fail and/or timeout, so run this command again on each manager to get all of the components to trust the old root again (still used in the 'ucp-node-certs' volume):

    for VOL in $ALL_CERT_VOLS; do
        docker run -i --rm -v $VOL:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/ca.pem' << EOF
        $OLD_ROOT_CERT
        $NEW_ROOT_CERT
        EOF
    done
    docker restart $UCP_CONTAINERS
    
  10. At this point, all of the managers should appear healthy again - all of their components should trust both the old and new cluster CA. Each cert is now signed by the new cluster CA except for those in the ucp-node-certs volume. To trigger this update, run these commands on each node:

    docker run --rm -v ucp-node-certs:/pki --entrypoint rm $UCP_AGENT_IMG /pki/cert.pem
    docker rm -f ucp-proxy
    
  11. Wait for all of the nodes to appear healthy again. At this point, there should be no certificates left in the system which are issued by the old cluster Root CA. Now it is safe to un-trust the old Root CA by running these commands on every manager node in the cluster:

    for VOL in $ALL_CERT_VOLS; do
        docker run -i --rm -v $VOL:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/ca.pem' << EOF
        $NEW_ROOT_CERT
        EOF
    done
    docker restart $UCP_CONTAINERS
    

Rotate UCP Client Root CA

Once the UCP cluster root CA has been successfully rotated, it is safe to rotate the UCP client root CA (the one used to issue UCP client certificates to non-admin users).

  1. Generate a new Root CA Key:

    openssl ecparam -name prime256v1 -genkey -noout -out new_client_key.pem
    export NEW_CLIENT_KEY=$(cat new_client_key.pem)
    
  2. Generate a new root CA cert (Enter . for most of the options (country name, organization, etc.) but use UCP Client Root CA for the Common Name.

    openssl req -new -x509 -key new_client_key.pem  -out new_client_cert.pem -days 3650
    export NEW_CLIENT_CERT=$(cat new_client_cert.pem)
    
  3. The following commands will replace the contents of the ucp-client-root-ca volume with the newly generated client root key and certificate, remove the UCP controller's cluster certificate (to trigger regeneration of its certificates again), and then remove the client root CA and ucp controller containers (to trigger the reconciliation process). Run these commands on each manager node:

    docker run -i --rm -v ucp-client-root-ca:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/cert.pem' << EOF
        $NEW_CLIENT_CERT
    EOF
    docker run -i --rm -v ucp-client-root-ca:/pki --entrypoint sh $UCP_AGENT_IMG -c 'cat > /pki/key.pem' << EOF
        $NEW_CLIENT_KEY
    EOF
    docker run --rm -v ucp-controller-client-certs:/pki --entrypoint rm $UCP_AGENT_IMG /pki/cert.pem
    docker rm -f ucp-client-root-ca ucp-controller