How do I recover pods in CrashLoopBackOff state?

Purpose

When a container in a pod fails repeatedly, Kubernetes sets the pod to CrashLoopBackOff and increases the delay between restart attempts. An NSP pod in this state is listed with a STATUS of CrashLoopBackOff, as shown below:

namespace pod_name 0/1 CrashLoopBackOff 5 12m

Use this procedure to inspect container logs and recover when logs report exec format error, which indicates that the container image on the node hosting the pod is corrupted. Scaling the workload to zero replicas and removing the corrupted image from the node allows Kubernetes to pull a fresh image when the workload is scaled back up.

Other log messages may indicate configuration errors, missing secrets, or registry image-pull failures. For those cases, collect the logs and contact Nokia support before taking corrective action beyond log collection. If a registry or TLS certificate problem is suspected, see How do I update the Kubernetes registry TLS certificate?.

Note: To list all pods or view pod events, see How do I view the status of all Kubernetes pods? and How do I retrieve pod information?.

Steps


1	Log in as the root or NSP admin user on the NSP deployer host and enter the following: `#` `export KUBECONFIG=/opt/nsp/nsp-configurator/kubeconfig/nsp_kubeconfig ↵`
2	Open a console window.
3	Enter the following to list the pods in the NSP cluster: `#` `kubectl get pods -A ↵` Identify the pod that has a STATUS value of CrashLoopBackOff and record its namespace and name.
4	Enter the following to identify the cluster node that hosts the pod: `#` `kubectl get pod pod_name -n namespace -o wide ↵` where pod_name is the name of the pod in CrashLoopBackOff state and namespace is the pod namespace Record the NODE value; you require this node when you remove the corrupted container image.
5	Enter the following to view the container logs for the pod: `#` `kubectl logs pod_name -n namespace ↵` If the pod has multiple containers, Kubernetes may default to a sidecar container and display a message such as the following: `Defaulted container "container_a" out of: container_a, container_b, container_c (init)` Enter the following to view logs for the failing container: `#` `kubectl logs pod_name -n namespace -c container_name ↵` where container_name is the name of the container that is crashing Review the log output to determine the cause of the failure. For example, the following output indicates a corrupted container image on the node: `exec /binary: exec format error` Note: If the log output does not contain exec format error, contact technical support with the log output before you continue.
6	If the log output contains exec format error, scale the workload that manages the pod to zero replicas. On the NSP deployer host, do one of the following: If the pod is managed by a StatefulSet, enter the following: `#` `kubectl scale statefulset workload_name -n namespace --replicas=0 ↵` where workload_name is the name of the StatefulSet If the pod is managed by a Deployment, enter the following: `#` `kubectl scale deployment workload_name -n namespace --replicas=0 ↵` where workload_name is the name of the Deployment The workload stops and the pod is terminated.
7	Log in as the root user on the NSP cluster node recorded in the NODE column.
8	Open a console window on the cluster node.
9	Enter the following to list the container images on the node: `#` `crictl images ↵` Identify the image ID of the image used by the pod that was in CrashLoopBackOff state.
10	Enter the following to remove the corrupted image: `#` `crictl rmi image_id ↵` where image_id is the image ID from the previous step. To retrieve the container image ID, run `#` `kubectl describe pod pod-name -n namespace`
11	Enter the following to remove unused images: `#` `crictl rmi --prune ↵` Unused images are removed from the node.
12	Close the console window on the cluster node.
13	Log in as the root or NSP admin user on the NSP deployer host and enter the following: `#` `export KUBECONFIG=/opt/nsp/nsp-configurator/kubeconfig/nsp_kubeconfig ↵`
14	Open a console window.
15	Scale the workload back to one replica. Do one of the following: If the pod is managed by a StatefulSet, enter the following: `#` `kubectl scale statefulset workload_name -n namespace --replicas=1 ↵` If the pod is managed by a Deployment, enter the following: `#` `kubectl scale deployment workload_name -n namespace --replicas=1 ↵` Kubernetes recreates the pod and pulls a fresh container image from the registry.
16	Enter the following periodically until the pod STATUS value is Running: `#` `kubectl get pods -A ↵` The pods are listed.
17	Close the console window. End of steps