Skip to main content

Docker Success Center

The Docker enterprise customer portal.

Docker, Inc.

Instrumented UCP HRM Testing Procedure

The following procedure outlines the first stage of isolating a communication problem between ucp-hrm and a backend service running on a given host.  The process involves pinging between ucp-hrm and the backend while collecting tcpdump along the network path: 1) inside the backend container and 2) at the ucp-hrm and backend hosts.

This procedure expects to be run with a UCP Admin client bundle loaded.

  1. Initialize some variables for later use.  Populate the variables BACKEND_SERVICE_NAME with the name of the backend service, BACKEND_HOSTS_NAME with the name of the desired host where a backend service replica is running, and UCP_NETWORK_NAME with the name of the HRM network (defaults is ucp-hrm).

    BACKEND_HOSTS_IP=$(docker node inspect --format {{.Status.Addr}} $BACKEND_HOSTS_NAME)
    BACKEND_CONTAINER_NAME=$(docker ps -f name=$BACKEND_SERVICE_NAME --format {{.Names}} |grep $BACKEND_HOSTS_NAME | head -n1)
    BACKEND_CONTAINER_IP=$(docker inspect --format "{{with index .NetworkSettings.Networks \"$UCP_NETWORK_NAME\"}}{{.IPAddress}}{{end}}" $BACKEND_CONTAINER_NAME)
    BACKEND_CONTAINER_NETNS=$(basename $(docker inspect --format {{.NetworkSettings.SandboxKey}} $BACKEND_CONTAINER_NAME))
    FRONTEND_HOSTS_ID=$(docker inspect --format '{{.NodeID}}' $(docker service ps $FRONTEND_SERVICE_NAME -f desired-state=running -q | head -n1))
    FRONTEND_HOSTS_NAME=$(docker node ls --format {{.Hostname}} -f id=$FRONTEND_HOSTS_ID)
    FRONTEND_HOSTS_IP=$(docker node inspect --format {{.Status.Addr}} $FRONTEND_HOSTS_NAME)
    FRONTEND_CONTAINER_NAME=$(docker ps -af name=ucp-hrm -f status=running --format {{.Names}} |head -n1)
    FRONTEND_CONTAINER_NETNS=$(basename $(docker inspect --format {{.NetworkSettings.SandboxKey}} $FRONTEND_CONTAINER_NAME))
  2. Start a tcpdump on the backend container filtering for ICMP traffic.

    docker run -d --rm --name netshoot_backend_tcpdump --privileged -e constraint:node==$BACKEND_HOSTS_NAME -v /var/run/docker/netns:/netns  nicolaka/netshoot nsenter --net=/netns/$BACKEND_CONTAINER_NETNS tcpdump -pni any icmp
  3. Start a tcpdump on the ucp-hrm host filtering for encrypted traffic towards the backend host.

    docker run -d --rm --name netshoot_frontend_host_tcpdump --net host -e constraint:node==$FRONTEND_HOSTS_NAME nicolaka/netshoot tcpdump -pni any esp and host $FRONTEND_HOSTS_IP and host $BACKEND_HOSTS_IP
  4. Start a tcpdump on the host running the backend service filtering for encrypted traffic from the ucp-hrm host.

    docker run -d --rm --name netshoot_backend_host_tcpdump --net host -e constraint:node==$BACKEND_HOSTS_NAME nicolaka/netshoot tcpdump -pni any esp and host $FRONTEND_HOSTS_IP and host $BACKEND_HOSTS_IP
  5. Initiate a continuous ping from the ucp-hrm container to the backend container IP.

    docker run -d --rm --name netshoot_pinger --privileged -e constraint:node==$FRONTEND_HOSTS_NAME -v /var/run/docker/netns:/netns  nicolaka/netshoot nsenter --net=/netns/$FRONTEND_CONTAINER_NETNS ping $BACKEND_CONTAINER_IP
  6. Confirm ping is failing (either no output or a stream of errors).

    docker logs --tail 10 -f netshoot_pinger
  7. Observe traffic inside the backend container.

    • Are ICMP echo requests received?

    • Are ICMP echo replies sent?

    docker logs --tail 10 -f netshoot_backend_tcpdump
  8. Observe traffic on the ucp-hrm host.

    • Is traffic unidirectional or bidirectional?

    docker logs --tail 10 -f netshoot_frontend_host_tcpdump
  9. Observe traffic on the backend host.

    • Is traffic unidirectional or bidirectional?

    docker logs --tail 10 -f netshoot_backend_host_tcpdump
  10. Cleanup.

    docker rm -f $(docker ps -f name=netshoot --format {{.Names}})