0 0 Share PDF

Cloud plugins such as Cloudstor not applying labels as expected

Article ID: KB000956

Issue

Label are not applied correctly when creating a volume through UCP with a volume storage plugin such as Cloudstor (Azure or AWS) or Netapp volume driver, resulting in a label mismatch when attempting to deploy a stack.

Sample volume create (command issued from a shell with a UCP client certificate bundle configured):

docker volume create --driver cloudstor:latest --label "com.docker.ucp.access.label=/infra" test_volume

When subsequently inspecting this volume, labels are found to be empty (null) instead of including the expected infra collection label specified in the previous step. Prepend the hostname to the volume name (hostname/volume) when inspecting volumes through a client bundle.

$ docker volume inspect dci-36iv65i2-manager-linux-1/test_volume
[
    {
        "Driver": "cloudstor:latest",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/plugins/26ae4ad90bb4e9c306dd0e0a970cccb573e4c125f075d4c31e8f53d5d87c481e/rootfs/mnt/cloudstor/test_volume",
        "Name": "test_volume",
        "Options": {},
        "Scope": "local"
    }
]

When attempting to deploy or update a stack with this volume, labels will not be applied correctly causing the below error:

failed to create service test: Error response from daemon: located 1 sets of existing volumes named "test_volume" with different collection labels. Please make sure all existing volumes with the same name share the same collection or don't have one at all

Prerequisites

Before performing these steps, you must meet the following requirements:

  • Using shared cloud storage plugin on Docker for Azure / Docker for AWS such as Cloudstor or Netapp Storage Driver

Resolution

To workaround the cloud storage plugin mismatch labels, please see the below steps. Please note this will only work for NEWLY created volumes. The quick summary is that the ucp-config.toml will be pulled down and the value of local_volume_collection_mapping will be changed from false to true. Below steps are for 3.0.X however document is listed below for how to pull the ucp-config.toml from 3.1.X versions under UCP 3.1.X below step 5.

Note: Even with local_volume_collection_mapping enabled, collection labels will still not appear in docker volume inspect. Instead, UCP stores volume collection information in ucp-kv.

UCP 3.0.X

How to pull/edit/push ucp-config.toml in 3.0.X

  1. View current ucp-config.toml and output to file to be edited:

    # CURRENT_CONFIG_NAME will be the name of the currently active UCP configuration
    CURRENT_CONFIG_NAME=$(docker service inspect ucp-agent --format '{{range .Spec.TaskTemplate.ContainerSpec.Configs}}{{if eq "/etc/ucp/ucp.toml" .File.Name}}{{.ConfigName}}{{end}}{{end}}')
    # Collect the current config with `docker config inspect`
    docker config inspect --format '{{ printf "%s" .Spec.Data }}' $CURRENT_CONFIG_NAME > ucp-config.toml
    
  2. Using text editor of choice, edit the ucp-config.toml to change the local_volume_collection_mapping value to true. Desired output after configuration change below:

    $ vim ucp-config.toml
    # After configuration change, new value confirmed below
    $ grep -i mapping ucp-config.toml
    local_volume_collection_mapping = true
    
  3. After editing the file, make new ucp-config.toml and use docker service update to apply the configuration from the edited file.

    # NEXT_CONFIG_NAME will be the name of the new UCP configuration
    NEXT_CONFIG_NAME=${CURRENT_CONFIG_NAME%%-*}-$((${CURRENT_CONFIG_NAME##*-}+1))
    # Create the new swarm configuration from the file ucp-config.toml
    docker config create $NEXT_CONFIG_NAME  ucp-config.toml
    # Use the `docker service update` command to remove the current configuration
    # and apply the new configuration to the `ucp-agent` service.
    docker service update --config-rm $CURRENT_CONFIG_NAME --config-add source=$NEXT_CONFIG_NAME,target=/etc/ucp/ucp.toml ucp-agent
    
  4. With the new ucp-config.toml loaded, create new volume and redeploy / update stack with newly created volume.

UCP 3.1.X

  1. Please follow the above procedure to edit the value in ucp-config.toml but use the below document on how to pull and edit the ucp-config.toml file for 3.1.X Version:

How to pull/edit/push ucp-config.toml in 3.1.X

What's Next

Docker for Azure persistent data volumes

Docker for AWS persistent data volumes