0 0 Share PDF

Kubelet is unhealthy following service network restart

Article ID: KB000872

Issue

Various container networking issues occur after service network restart with worker nodes showing as Down with message Kubelet is unhealthy: Kubelet stopped posting node status.

The following examples show an affected node as examined through a UCP client certificate bundle:

$ docker node ls |egrep '(^ID|Down)'
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
jkaqgwiac07f8aaa4g2r9pslm     worker-1            Down                Active                                  17.06.2-ee-16
$ docker node inspect worker-1 --format '{{.Status.Message}}'
Kubelet is unhealthy: Kubelet stopped posting node status., Kubelet stopped posting node status., Kubelet stopped posting node status.
$ kubectl get nodes | egrep '(^NAME|NotReady)'
NAME       STATUS     ROLES     AGE       VERSION
worker-1   NotReady   <none>    7d        v1.8.11-docker-8d637ae

Prerequisites

Before performing these steps, you must meet the following requirements:

  • Docker EE 2.0
  • Docker Universal Control plane 3.0
  • CentOS or Red Hat Enterprise Linux 7

Root Cause

IP forwarding is required for container networking to function.

dockerd enables ip forwarding (sysctl net.ipv4.ip_forward) when it starts.

service network restart disables ip forwarding while stopping networking.

Resolution

Perform the following commands from a shell session on affected nodes:

  1. Confirm the node is affected:

    docker info >/dev/null
    

    If affected, the output will contain WARNING: IPv4 forwarding is disabled

  2. Re-enable ip forwarding via sysctl net.ipv4.ip_forward:

    sudo sysctl -w net.ipv4.ip_forward=1
    

    Optionally net.ipv4.ip_forward=1 can be enabled independently of dockerd at the system level, so that service network restart does not clear this setting.

    echo "net.ipv4.ip_forward = 1" | sudo tee -a /etc/sysctl.conf
    
  3. The node will come back as healthy.