0 0 Share PDF

Docker Reference Architecture: Universal Control Plane Service Discovery and Load Balancing for Kubernetes

Article ID: KB001031

Introduction

Service discovery is used to register a service and publish its connectivity information so that other services (both within and outside of the cluster) are aware of how to connect to the service. As applications move toward microservices and service-oriented architectures, service discovery has become an integral part of any distributed system, increasing the operational complexity of these environments.

Docker Enterprise (DE) includes Kubernetes (k8s for short) out of the box allowing users to take advantage of its full potential out-of-the-box. The Kubernetes service included in Docker Enterprise is referred to as Docker Kubernetes Service (DKS). By default, DKS comes with some service discovery and load balancing capabilities to aid the DevOps initiatives across any organization. In this reference architecture, we will walk you through key service discovery and load balancing capabilities in DKS along with recommended best practices on their usage.

For Swarm-based service discovery and load-balancing, please refer to this reference architecture.

Assumptions

This article assumes you have a basic understanding of Docker Enterprise and some Kubernetes fundamentals such as containers, pods, container images, YAML configurations, deployments, and kubectl usage.

What You Will Learn

This reference architecture covers the solutions that Docker Enterprise 3.0 (or newer) provides in the topic areas of service discovery and load balancing for Kubernetes workloads.

Versions

This document pertains to Docker Enterprise 3.0 (or newer) which includes UCP version 3.2 (or newer), Kubernetes version 1.14.6, and Calico version 3.8.2.

Kubernetes Services

When you deploy an application in DKS using the Deployment object or directly through the Pod objects, your application pod(s) will be scheduled to run on your cluster. Each pod will have an internal IP address that can be used to reach it within the cluster. Typically, connecting to individual pods directly is a bad idea as pods are ephemeral and are regularly spinning up or down. Kubernetes has a long-lived Service construct to address this challenge. By associating a set of pods to a service, you can start connecting to your application using the service (by DNS name or IP). The service will do the work of forwarding traffic to healthy pods. For example, here is a deployment object definition for a sample application along with its service definition:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: dockerdemo-deploy
  labels:
    app: dockerdemo-deploy
spec:
  selector:
    matchLabels:
      app: dockerdemo
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: dockerdemo
    spec:
      containers:
      - image: ehazlett/docker-demo
        name: docker-demo-container
        env:
        - name: app
          value: dockerdemo
        ports:
        - containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: docker-demo-svc
spec:
  selector:
    app: dockerdemo
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080

More info on Kubernetes services can be found here

Service Discovery

Kubernetes supports two primary modes of finding a Service, environment variables(injected by the kubelet when pods are created) and DNS. Both are valid options, however, the most common method to discover and reach a service from within DKS is to use the embedded DNS service.

Service Discovery with Environment Variables

When a pod gets spun up, the kubelet injects a set of environment variables into it. The name of all configured services and their associated ports within the same namespace are injected in every pod. These environment variables can then be used by the application to connect to these services. Below is an example of the env output to a pod. Note that there is a SQL Server service deployed within the same namespace.

/ # env
app=dockerdemo
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.96.0.1:443
SQLSERVER_SERVICE_PORT=1433
SQLSERVER_PORT_1433_TCP=tcp://10.96.221.251:1433
SQLSERVER_PORT=tcp://10.96.221.251:1433
SQLSERVER_SERVICE_HOST=10.96.221.251

If your application expects environment variables to know about some services it depends on, you can use this method. However, this is not dynamically updated in the case of the removal/redeployment of services.

Service Discovery with DNS

DKS comes with a highly-available DNS service using kube-dns. DKS will be moving to CoreDNS in the next major release given that it's a more performant and CNCF-graduated project.

Like any other service, the kube-dns service is also assigned a VIP. By default it's 10.96.0.10 and gets injected into every pod's /etc/resolv.conf as a nameserver by the node's kubelet. Please see below:

🐳  → kubectl get svc kube-dns -n kube-system -o wide
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE   SELECTOR
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   30d   k8s-app=kube-dns
## FROM INSIDE THE POD
/ # cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local us-west-2.compute.internal
options ndots:5

By default, all services are assigned a DNS A record for a name of the form <SERVICE-NAME>.<NAMESPACE>.svc.cluster.local. This record will be associated with a Virtual IP (VIP) for the service. Any other applications within the cluster, across all namespaces, should be able to resolve this service using its DNS name. If the application pod trying to resolve this service also lives within the same namespace (e.g default namespace) then it should be able to resolve it using <SERVICE-NAME> because the name resolution default to .<NAMESPACE>.svc.cluster.local. Otherwise, the namespace needs to be included in the DNS request.

More info on Kubernetes DNS for Services and Pods is here

Exposing Services

Applications typically need a mechanism to be accessed in some way. Sometimes they are only accessed from within the cluster. In other situations, they are accessed from external clients or other clusters. In the next section, we'll go through the available options for exposing your services: ClusterIP, NodePort, LoadBalancer. These types can be defined as part of the service's YAML configuration using the Type option.

ClusterIP

Services will receive a cluster-scoped Virtual IP address also known as ClusterIP. This IP is reserved from a separate subnet dedicated for services IP allocation(service-cluster-ip-range). This IP will be reachable from all namespaces within the cluster. This is the default method of exposing the service internally. Once the service is created and a VIP is associated with it, every kube-proxy running on every cluster node will program an iptables rule so that all traffic destined to that VIP will be redirected to one of the Service’s backend pods instead.

By default, and unless otherwise configured, the services subnet is 10.96.0.0/16 and the pods subnet is 192.168.0.0/16. These configurations can be customized using the --service-cluster-ip-range and --pod-cidr install options, respectively, only during the initial provisioning of the UCP cluster. More info on these customization can be found here.

Below is an example of a Service definition that uses ClusterIP.

apiVersion: v1
kind: Service
metadata:  
  name: my-service
spec:
  selector:    
    app: my-app
  type: ClusterIP
  ports:  
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

If the service needs to be accessed from outside the cluster, then there are a couple of available options to doing so. Mainly NodePort and LoadBalancer. Ingress is another method that you can use, and we will discuss. Ingress is technically not a Type of a service, but it's another method for using Layer 7 based routing to expose your services externally.

NodePort

If you use NodePort service type, a random and available TCP or UDP port will be allocated from the port range (default is "32768-35535" in DKS) of the node itself. Each node proxies that port (the same port number on every node) into your Service. If you want to specify the port that the node should listen on you can do that under the Ports section in the YAML config of the Service. If the port is already in-use, then the deployment will fail and you will see an error describing that. Here's an example of a service with defined ports:

apiVersion: v1
kind: Service
metadata:  
  name: my-service
spec:
  selector:    
    app: my-app
  type: NodePort
  ports:  
  - name: http
    port: 8080
    targetPort: 80
    protocol: TCP

When this Service is deployed, all nodes in the DKS cluster will listen on port 8080 and forward to port 80 on the back-end pods.

This service type is well suited for application services that require direct TCP/UDP access, like RabbitMQ for example. If your service can be accessed using HTTP/HTTPs then Ingress is recommended (we will discuss Ingress in the following sections).

Final note, when you create a service with Type: NodePort, the service will still acquire a cluster IP to be used for internal cluster traffic.

LoadBalancer

If you are running Docker Enterprise on AWS or Azure, this service type allows you to integrate natively with the underlying cloud's load balancing services by provisioning a load balancer for your Service and configuring it to forward traffic directly to your service. Although this can be useful in some cases, provisioning a dedicated external loadbalancer for your Kubernetes services can be an overkill. Below is an example of a service that utilizes AWS's ELB integration:

 🐳  → cat service-lb.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: docker-demo-svc-lb
spec:
  selector:
    app: dockerdemo
  ports:
    - port: 8080
      targetPort: 8080
  type: LoadBalancer

🐳  → kubectl get svc
NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)          AGE
docker-demo-svc-lb             LoadBalancer   10.96.0.69      XXXXXXXXX.us-west-2.elb.amazonaws.com   8080:35494/TCP   35m

Note: This integration only works with AWS and Azure and requires some initial setup during UCP installation, please see this for more details.

More details on exposing services and service types is here

The Kubernetes Layer 7 Routing (Ingress)

Ingress is a Kubernetes API object that manages external access to the services in a cluster, typically HTTP/HTTPS. Users can define the ingress objects as part of their deployment YAML configurations. However, for the actual ingress functionality to work properly, you need to deploy an Ingress Controller separately in UCP. The ingress controller is the control-plane for wiring up external addresses(e.g a L7 URL like app.example.com) to a service deployed within the cluster.

There are certainly many options available when it comes to Ingress controllers. We recommend that you use one of the two deployment options listed below. Please note that neither of these options is currently officially supported by Docker.

Option A: Istio-based Cluster Ingress (Experimental in UCP 3.2)

Cluster Ingress provides layer 7 services to traffic entering a Docker Enterprise cluster for a variety of different use-cases that help provide application resilience, security, and observability. Ingress offers dynamic control of L7 routing in a highly available architecture that is also highly performant. DKS's Ingress for Kubernetes is based on the Istio control-plane and is a simplified deployment focused on just providing ingress services with minimal complexity, including features such as:

  • L7 host and path routing

  • Complex path matching and redirection rules

  • Weight-based load balancing

  • TLS termination

  • Persistent L7 sessions

  • Hot config reloads

    Note: The next major release of UCP will include a built-in and supported version of this functionality out-of-the-box.

Full details on deploying this solution can be found here

Option B: NGINX Ingress Controller

NGINX Ingress controller is a widespread and production-ready solution that provides layer-7 routing functionality with ease. Although Docker does not officially support it, it had been tested to work with DKS. The following guide walks you through an end-to-end production deployment of this solution.

Once you deploy the NGINX Ingress controller, you can configure Ingress objects/routes for your application. For example, here is an ingress object docker-demo-svc service we described earlier:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dockerdemo-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: dockerdemo.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: docker-demo-svc
          servicePort: 80

This specific example uses the NGINX ingress controller, indicate in the annotations section. The spec section is used to associate the L7 URL/host to the backend service.

Finally, once you deploy the ingress object, you can see your ingress routes under the Ingress section within UCP:

img-ingress

Detailed documentation on Ingress configurations can be found here

Summary

The ability to scale and discover services in Docker Enterprise is now easier than ever. With the service discovery and load balancing features built into Docker Kubernetes Service, developers can spend less time creating these types of supporting capabilities on their own and more time focusing on their applications.