0 0 Share PDF

DTR 2.6 lost tags after reconfiguring storage

Article ID: KB000954

Issue

After reconfiguring Docker Trusted Registry (DTR) image storage there are no tags visible under the Repositories page within the WebUI. docker push and docker pull may also be failing.

Prerequisites

In order to be affected by this issue, you must meet the following prerequisites:

  • Docker Trusted Registry 2.5, with Experimental OnlineGC Enabled
  • Docker Trusted Registry versions 2.6.0 through 2.6.3
  • Attempting to reconfigure registry image storage

Root Cause

DTR purges tag metadata when reconfiguring image storage. In earlier versions, DTR would subsequently start a tagmigration job to rebuild tag metadata from the file layout in the image layer store.

The image store layout was changed to a content addressable format to support online garbage collection. Online garbage collection was introduced as an experimental feature in DTR 2.5 starting with version 2.5.4 and fully implemented in DTR 2.6 as the default method for garbage collection starting with version 2.6.0. The content addressable layout does not contain tag information, so tag metadata cannot be rebuilt from the image store following a storage reconfiguration.

Internal improvement request ENGDTR-793 is tracking development of an enhancement to enable the preservation of tag metadata during storage reconfiguration in DTR 2.6. As of 7 March 2019, this feature was targeted for release with DTR 2.6.4.

Resolution

Use the following table to identify the recommended recovery strategy for your scenario. Details for each strategy are listed below and can be reached by clicking on the name of the strategy in the right hand column.

Attempting Reconfigure from Attempting Reconfigure to DTR Still Exists? Recoverable to Recommended Reconfiguration or Recovery Method
NFS1 NFS1 yes original or new NFS1 reconfigure --dtr-storage-volume
NFS1 NFS1 no original or new NFS1 restore --dtr-storage-volume
NFS1 cloud2 yes NFS1 reconfigure --dtr-storage-volume
NFS1 cloud2 no NFS1 restore --dtr-storage-volume
cloud2 NFS1 yes cloud2 restore --dtr-use-default-storage
cloud2 NFS1 no cloud2 restore --dtr-use-default-storage
cloud2 cloud2 yes original cloud2 restore --dtr-use-default-storage
cloud2 cloud2 no original cloud2 restore --dtr-use-default-storage

1 change of nfs server, export path, or mount options via dtr reconfigure --nfs-* flags

2 Amazon S3 or compatible, Google Cloud Storage, Microsoft Azure Blob storage, OpenStack Swift

Where:

  • Attempting Reconfigure from: indicates the type of storage you are attempting to reconfigure from.
  • Attempting Reconfigure to: indicates the type of storage you are attempting to reconfigure to.
  • DTR Still Exists?: indicates whether or not DTR is still installed, or if you are in a disaster recovery scenario.
  • Recoverable to: indicates the type of storage you can recover to.
  • Recommended Reconfiguration or Recovery Method: indicates the preferred recovery strategy for the given scenario.

Reconfigure Using a Local NFS Volume

If you are reconfiguring from NFS and DTR is still installed, you can recover from this issue by creating a local NFS volume on each DTR node and then use DTR reconfigure to reference the new volume. Although not required, we recommend you take a DTR metadata backup prior to performing this step.

  1. On each DTR node, create a local volume pointing to the desired NFS export. Use the same volume name on each node. nfs-host and path in the following commands can reference any NFS export, as long as the export contains the image data. This step uses the following example variables:

    • nfs-host: NFS server that holds your blob data
    • path: directory path on the server
    • dtr-registry-admin-lvol-nfs: volume name. You can use whatever name you like, but we recommend something with nfs in the name to indicate this is an NFS volume.

    To create the local volume using NFSv3:

    docker volume create -o type=nfs \
      -o o=addr=nfs-host,rw,sync,actimeo=0 \
      -o 'device=:/path' \
      dtr-registry-admin-lvol-nfs
    

    To create the local volume using NFSv4:

    docker volume create -o type=nfs \
      -o o=addr=nfs-host,rw,nfsvers=4,async \
      -o 'device=:/path' \
      dtr-registry-admin-lvol-nfs
    

    See docker volume create documentation for more details.

  2. Validate the contents of the volume by confirming the image store directory layout is present:

    docker run -it --rm \
      -v dtr-registry-admin-lvol-nfs:/volume \
      alpine \
      ls /volume/docker/registry/v2/{repositories,blobs}
    
  3. Use docker/dtr reconfigure --dtr-storage-volume to configure DTR to use the new volume.

    docker run --rm -it \
      docker/dtr:2.6.2 reconfigure \
      --ucp-url ucp.example.com \
      --ucp-username admin \
      --ucp-insecure-tls \
      --dtr-storage-volume dtr-registry-admin-lvol-nfs
    

Note: This approach introduces a requirement for the named volume to be present on nodes that may be added to the DTR cluster in the future using the docker/dtr join command. If the volume does not exist on a node, you will need to create the volume prior to joining the node.

Restore to a Local NFS Volume

If you are reconfiguring from NFS and you are in a disaster recovery scenario, you can recover from this issue by creating a local NFS volume on each DTR node, ensuring that DTR has been completely uninstalled, and then using DTR restore to reference the new volume. This strategy requires a valid DTR metadata backup.

  1. On each DTR node, create a local volume pointing to the desired NFS export. Use the same volume name on each node. nfs-host and path in the following commands can reference any NFS export, as long as the export contains the image data. This step uses the following example variables:

    • nfs-host: NFS server that holds your blob data
    • path: directory path on the server
    • dtr-registry-admin-lvol-nfs: volume name. You can use whatever name you like, but we recommend something with nfs in the name to indicate this is an NFS volume.

    To create the local volume using NFSv3:

    docker volume create -o type=nfs \
      -o o=addr=nfs-host,rw,sync,actimeo=0 \
      -o 'device=:/path' \
      dtr-registry-admin-lvol-nfs
    

    To create the local volume using NFSv4:

    docker volume create -o type=nfs \
      -o o=addr=nfs-host,rw,nfsvers=4,async \
      -o 'device=:/path' \
      dtr-registry-admin-lvol-nfs
    

    See docker volume create documentation for more details.

  2. Validate the contents of the volume by confirming the image store directory layout is present:

    docker run -it --rm \
      -v dtr-registry-admin-lvol-nfs:/volume \
      alpine \
      ls /volume/docker/registry/v2/{repositories,blobs}
    
  3. Make sure that DTR is uninstalled using the docker/dtr destroy command until there are no replicas left in the cluster:

    docker run --rm -it \
      docker/dtr:2.6.2 destroy \
      --ucp-url ucp.example.com \
      --ucp-username admin \
      --ucp-insecure-tls
    
  4. Use docker/dtr restore --dtr-storage-volume to restore the cluster using the new local volume.

    read -sp "UCP password: " UCP_PASSWORD
    docker run -i --rm \
      --env UCP_PASSWORD=$UCP_PASSWORD \
      docker/dtr:2.6.2 restore \
      --ucp-url ucp.example.com \
      --ucp-insecure-tls \
      --ucp-username admin \
      --ucp-node hostname \
      --dtr-external-url dtr.example.com \
      --dtr-storage-volume dtr-registry-admin-lvol-nfs < dtr-metadata-backup.tar
    

Note: This approach introduces a requirement for the named volume to be present on nodes that may be added to the DTR cluster in the future using the docker/dtr join command. If the volume does not exist on a node, you will need to create the volume prior to joining the node.

Restore to Cloud Storage

If you were attempting to reconfigure from cloud storage, you can recover from this issue by uninstalling and restoring DTR using --dtr-use-default-storage to re-use the original cloud storage. This strategy requires a valid DTR metadata backup.

  1. Make sure that DTR is uninstalled using the docker/dtr destroy command until there are no replicas left in the cluster:

    docker run --rm -it \
      docker/dtr:2.6.2 destroy \
      --ucp-url ucp.example.com \
      --ucp-username admin \
      --ucp-insecure-tls
    
  2. Use docker/dtr restore --dtr-use-default-storage to restore the cluster re-using the original cloud storage.

    read -sp "UCP password: " UCP_PASSWORD
    docker run -i --rm \
      --env UCP_PASSWORD=$UCP_PASSWORD \
      docker/dtr:2.6.2 restore \
      --ucp-url ucp.example.com \
      --ucp-insecure-tls \
      --ucp-username admin \
      --ucp-node hostname \
      --dtr-external-url dtr.example.com \
      --dtr-use-default-storage < dtr-metadata-backup.tar