Kubernetes Issues

Overview

This article lists a couple of issues that occurred while I was building this environment. The issues don’t fit into any of the specific articles mainly because it was likely due to my testing vs anything that occurred during the installation. The final article will be accurate and should just work however during the testing, these were identified and I had to track down a fix.

Terminating Pods

I had some pods that got stuck terminating for a long period of time. There were a couple of suggestions but the two that seemed to work best was to either force it or clear any finalizers for the pod. A finalizer is basically a task that needs to run and the pod is waiting for a successful completion of the task before it removes the pod.

The main solution I used for this was to force the delete. Make sure you know what node the pod is on before deleting it.

$ kubectl get pods -llamas -o wide
NAMESPACE            NAME                                                        READY   STATUS        RESTARTS      AGE    IP                NODE                                 NOMINATED NODE   READINESS GATES
llamas               llamas-f8448d86c-br4z8                                      1/1     Terminating   0             3d1h   10.42.232.197     tato0cuomknode3.stage.internal.pri   <none>           <none>
$ kubectl delete pod llamas-f8448d86c-br4z8 -n llamas --grace-period=0 --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "llamas-f8448d86c-br4z8" force deleted

This can be an issue as noted, the underlying resource the pod was waiting on could also be stuck. The main thing I did was to restart kubelet on the node the pod was running on. Make sure you get the node name before forcing the deletion. Otherwise it’s safest to restart kubelet on all worker nodes, no fun if there are more than a few. If you don’t have privileged access to the node though, you’ll have to get with your sysadmin team.

Application Deletion

In working with projects in ArgoCD, I mucked up one of the projects so badly it couldn’t be deleted. This was due to some process that needed to complete but had been removed outside the normal deletion process (apparently). This time I removed the finalizer process and the application simply was deleted.

$ kubectl get application -A
NAMESPACE   NAME     SYNC STATUS   HEALTH STATUS
argocd      llamas   Unknown       Unknown
$ kubectl patch application/llamas -n argocd --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
application.argoproj.io/llamas patched
$ kubectl get application -A
No resources found