0 0 Share PDF

How to Delete a Kubernetes Namespace Stuck in the Terminating State

Article ID: KB000964

Issue

After trying to delete a kubernetes namespace, it is stuck in the terminating state.

$ kubectl delete ns skynet
Error from server (Conflict): Operation cannot be fulfilled on namespaces "skynet": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system. 

$ kubectl describe ns skynet
Name: skynet 
Labels: <none> 
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"name":"skynet","namespace":""}} 

Status: Terminating 

Root Cause

Namespaces get stuck in the Terminating state for various reasons, which are generally problems removing some of the namespace's resources from the cluster.

Resolution

There are a few options for clearing a namespace stuck in Terminating, and the option you choose depends on your goals. Each has advantages and disadvantages. Also, future releases will make it easier to identify why this is happening.

  • Option 1: You can examine the cluster to try to identify why the namespace is stuck, and attempt to manually resolve the issue

  • Advantage: If the resources causing the issue are identified and manually cleaned up, the namespace can be deleted without leaving any dangling resources in the cluster.

  • Disadvantage: This method takes longer and doesn't have a guarantee of a quick resolution, and may require some further troubleshooting to identify the issues.

  • Option 2: You can get the stuck namespace to finish deleting by removing it's finalizers

  • Advantage: This is a quick solution to clearning the namespace

  • Disadvantage: Whatever resources which couldn't be automatically removed during the namespace deletion will not be identified and will remain in the cluster.

Upcoming patch in Kubernetes 1.16

The 1.16 release will include a new feature which provides visibility into the namespace lifecycle controller to reveal why namespace deletions fail. This should make it easier to troubleshoot and resolve this issue manually.

Option 1: Identify and manually remove resources which can't be automatically deleted

This option is less straightforward but is probably a better solution in most cases because it doesn't leave namespace resources in the cluster after the namespace is deleted.

  1. Check for namespace resources which aren't being automatically removed:

After attempting to delete the namespace, some resources may not have been deleted. If you can identify and manually remove all remaining resources the namespace will finish Terminating.

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -n namespace name

  1. Check for any apiservices which are unavailable and not serving their resources:

kubectl get apiservice | grep False

Option 2: Remove a namespace's finalizers to quickly clear the namespace stuck in "Terminating" state

The following steps will quickly clear a namespace stuck in the "Terminating" state, but may have the consequence of leaving resources which belonged to that namespace in the cluster because they couldn't be deleted automatically.

  1. Enable docker admin client bundle

  2. GET the namespace object

    $ NAMESPACE=skynet
    $ kubectl get ns $NAMESPACE -o json > /tmp/${NAMESPACE}.json
    $ cat /tmp/${NAMESPACE}.json
    {
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
    "creationTimestamp": "2019-02-04T16:15:06Z",
    "name": "skynet",
    "resourceVersion": "20879144",
    "selfLink": "/api/v1/namespaces/skynet",
    "uid": "0951bcf9-2898-11e9-8e38-0242ac11000b"
    },
    "spec": {
    "finalizers": [
    "kubernetes"
    ]
    },
    "status": {
    "phase": "Active"
    }
    }
    
  3. Use a text editor to delete the all elements in finalizer array, leaving only an empty array [] such as below example:

    {
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
    "creationTimestamp": "2019-02-04T16:15:06Z",
    "name": "skynet",
    "resourceVersion": "20879144",
    "selfLink": "/api/v1/namespaces/skynet",
    "uid": "0951bcf9-2898-11e9-8e38-0242ac11000b"
    },
    "spec": {
    "finalizers": [
    ]
    },
    "status": {
    "phase": "Active"
    }
    }
    
  4. PUT the namespace object without the finalizer

    a. Get a bearer token

    ```
    UCP_URL=ucp.example.com
    USERNAME=admin
    PASSWORD=supersecretadminpassword
    curl -sk -d "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" https://${UCP_URL}/auth/login | jq -r .auth_token > auth-token
    ```
    

    b. Issue an HTTP PUT

    `curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json https://kubernetes-cluster-ip/api/v1/namespaces/skynet/finalize`
    
    Example: `curl -k -H "Content-Type: application/json" -H "authorization: Bearer $(cat ./auth-token)" -X PUT --data-binary @/tmp/skynet.json https://ucp.example.com/api/v1/namespaces/skynet/finalize`
    

After the finalizer array is empty and the status is in terminating, kubernetes will delete the namespace.