How to Delete a Namespace Stuck in the Terminating State

Sun, 08 Oct 2023 18:48:17 +0800

How to Delete a Namespace Stuck in the Terminating State

Sometimes, after using kubectl delete ns xxx to delete a namespace, you may find that the namespace does not disappear and remains in the Terminating state for a long time. This may occur because the kubelet is blocked, indicating that other resources are still using the namespace. Restarting the kubelet and trying to delete it again may not work, and even forcing the deletion with the following command may fail.

1

kubectl delete ns <namespace-name> --force --grace-period=0

Solution

1. Find the Namespace in the Terminating State

1

kubectl get ns

2. Retrieve the Namespace Resource

Use the following command to retrieve the namespace resource file in JSON format, and inspect the finalizer property under spec.

1

kubectl get namespace <namespace-name> -o json

We can see the namespace information:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


{
 "apiVersion": "v1",
 "kind": "Namespace",
 "metadata": {
 "creationTimestamp": "2023-10-08T01:54:39Z",
 "deletionTimestamp": "2023-10-08T01:56:59Z",
 "labels": {
 "k8slens-edit-resource-version": "v1"
 },
 "name": "slxt-pro-copy-test",
 "resourceVersion": "362142903",
 "selfLink": "/api/v1/namespaces/slxt-pro-copy-test",
 "uid": "ed12052c-02f8-4974-9603-ae087cad1bd1"
 },
 "spec": {
 "finalizers": [
 "kubernetes"
 ]
 },
 "status": {
 ...
 "phase": "Terminating"
 }
}

3. Save the JSON File Locally

You can use the following command to save the file directly, or copy and save it manually.

1

kubectl get namespace <namespace-name> -o json >ns.json

After running the command, you will find a new file named ns.json locally. You may also use any other filename.

4. Edit `ns.json` and Set `finalizers` to an Empty Array

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
 "apiVersion": "v1",
 "kind": "Namespace",
 "metadata": {
 "creationTimestamp": "2023-10-08T01:54:39Z",
 "deletionTimestamp": "2023-10-08T01:56:59Z",
 "labels": {
 "k8slens-edit-resource-version": "v1"
 },
 "name": "slxt-pro-copy-test",
 "resourceVersion": "362142903",
 "selfLink": "/api/v1/namespaces/slxt-pro-copy-test",
 "uid": "ed12052c-02f8-4974-9603-ae087cad1bd1"
 },
 "spec": {
 "finalizers": []
 },
 "status": {
 "phase": "Terminating"
 }
}

5. Start the Proxy Using `kubectl`

1

kubectl proxy

By default, this command starts a proxy service on local port 8001 to proxy the Kubernetes cluster API. Once it is running, we can use the API endpoint on port 8001 to resolve this issue.

6. Call the API Using the Command

Open a new terminal window and run the following command:

1

curl -k -H "Content-Type: application/json" -X PUT --data-binary @ns.json http://127.0.0.1:8001/api/v1/namespaces/<namespace-name>/finalize

Replace <namespace-name> with the name of the namespace in the Terminating state. Run the command from the same directory where ns.json is located. In this example, it is the home directory.

The API call will output the following information:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 "kind": "Namespace",
 "apiVersion": "v1",
 "metadata": {
 "name": "slxt-pro-copy-test",
 "selfLink": "/api/v1/namespaces/slxt-pro-copy-test/finalize",
 "uid": "ed12052c-02f8-4974-9603-ae087cad1bd1",
 "resourceVersion": "362142903",
 "creationTimestamp": "2023-10-08T01:54:39Z",
 "deletionTimestamp": "2023-10-08T01:56:59Z",
 "labels": {
 "k8slens-edit-resource-version": "v1"
 }
 },
 "spec": {
 
 },
 "status": {
 "phase": "Terminating",
 ...
 }
}

7. Verify That the Namespace Has Been Deleted

1

kubectl get namespaces

Kubernetes on Jalen's blog

How to Delete a Namespace Stuck in the Terminating State

How to Delete a Namespace Stuck in the Terminating State

Solution

1. Find the Namespace in the Terminating State

2. Retrieve the Namespace Resource

3. Save the JSON File Locally

4. Edit ns.json and Set finalizers to an Empty Array

5. Start the Proxy Using kubectl

6. Call the API Using the Command

7. Verify That the Namespace Has Been Deleted

4. Edit `ns.json` and Set `finalizers` to an Empty Array

5. Start the Proxy Using `kubectl`