0 0 Share PDF

Kublet restarting after upgrade to Universal Control Plane 3.1

Article ID: KB000931

Issue

Universal Control Plane (UCP) manager nodes in down status after upgrade to UCP 3.1 with one of the following status messages:

  • Kubelet is unhealthy: container runtime is down
  • Kubelet is unhealthy: Kubelet stopped posting node status., Kubelet stopped posting node status., Kubelet stopped posting node status., Kubelet stopped posting node status.

ucp-kubelet container restarting every minute on manager nodes, with the following Fatal error pertaining to ephemeral-storage in its container logs:

F0204 20:05:14.746665 1559 kubelet.go:1333 Failed to start ContainerManager Invalid Node Allocatable configuration. Resource "ephemeral-storage" has an allocatable of {{4723415455 0} {} BinarySI}, capacity of {{-438933919 0} {} BinarySI}

/var/lib/kubelet has 4GiB or less available, according to df -h /var/lib/kubelet

Prerequisites

Before performing these steps, you must meet the following requirements:

  • Universal Control Plane (UCP) 3.1

Root Cause

UCP storage requirements for Kubernetes were increased on manager nodes in UCP 3.1.

UCP 3.1.0 Release Notes

The default Kubelet configuration for UCP Manager nodes is expecting 4GB of free disk space in the /var partition. See System Requirements for details. Ref: https://docs.docker.com/ee/ucp/release-notes/#version-31

Resolution

Increase the size of the /var/lib/kubelet filesystem to have at least 5GiB free space. 6GiB free space recommended.

  1. Determine the filesystem device and check the current utilization of /var/lib/kubelet:

    df -h /var/lib/kubelet
    
  2. Increase the size of the filesystem using appropriate operating system commands. The following example assumes the filesystem is mounted on a logical volume named /dev/mapper/vg0-var, and needs 4GiB additional to meet the recommended 6GiB:

    sudo lvresize -rL +4G /dev/mapper/vg0-var
    
  3. Confirm available space exceeds 5GiB:

    df -h /var/lib/kubelet
    
  4. Watch docker ps -f name=ucp-kubelet for kubelet to remain in an Up state for longer than a minute.

  5. Check node status in the UCP WebUI.

Emergency Workaround

If it is not immediately possible to increase the size of the /var/lib/kubelet filesystem, the kubelet ephemeral storage storage requirement can be temporarily reduced to by editing the UCP configuration.

Modify the UCP configuration setting manager_kube_reserved_resources in the [cluster_config] table to "cpu=1,memory=2Gi,ephemeral-storage=4Gi", where the default 4Gi is replaced with a smaller value that suits currently available storage.

What's Next