Upgrading Kubernetes Clusters
This document provides a guide to upgrading the Kubernetes clusters in the quickest manner. Much of the upgrade process can be done using Ansible Playbooks. There are a few processes that need to be done centrally on the tool server. And the OS and control plane updates are also manual in part due to the requirement to manually remove servers from the Kubernetes API pool.
In most cases, examples are not provided as it is assumed that you are familiar with the processes and can perform the updates without having to be reminded of how to verify.
For any process that is performed with an Ansible Playbook, it is assumed you are on the lnmt1cuomtool11 server in the /usr/local/admin/playbooks/cschelin/kubernetes directory. All Ansible related steps expect to start from that directory. In addition, the application of pod configurations will be in the configurations subdirectory.
Perform Upgrades
Patch Servers
In the 00-osupgrade directory, you’ll be running the master and worker scripts. I recommend opening two windows, one for master and one for worker, and running each script with master -t [tag] and worker -t [tag]. This will verify a node is Ready, drain the node from the pool if a worker, perform a yum upgrade and reboot, uncordon again if a worker, and verify the nodes are Ready again. Should a node fail to be ready in time, the script will exit.
Update Versionlock
In the 03-packages directory, run the update -t [tag] script. This will install yum-plugin-versionlock if missing, remove old versionlocks, create new versionlocks for kubernetes, kubernetes-cni, and docker, and then the components will be upgraded.
Upgrade Kubernetes
Using the kubeadm command on the first master server, upgrade the first master server.
# kubeadm upgrade apply v1.20.6Update Control Planes
On the second and third master, run the kubeadm upgrade apply v1.20.6 command and the control plane will be upgraded.
Update kube-proxy
Check the kube-proxy daemonset and update the image tag if required.
kubectl edit daemonset kube-proxy -n kube-system- Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000
Save the changes.
Update coredns
Check the coredns deployment and update the image tag if required.
kubectl edit deployment coredns -n kube-system- Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000
Save the changes.
Restart kubelet and docker
In the 04-kubelet directory, run the update -t [tag] script. This will restart kubelet and docker on all servers.
Calico Upgrade
In the configurations/calico directory, run the following command:
$ kubectl apply -f calico.yamlcalicoctl Upgrade
Pull the updated calicoctl binary and copy it to /usr/local/bin. It’s likely already there but verify.
$ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.18.2/calicoctlkube-state-metrics Upgrade
In the configurations directory, /kube-state-metrics directory, run the following command:
$ kubectl apply -f kube-state-metrics/
metrics-server Upgrade
In the configurations/metrics-server directory, run the following command:
$ kubectl apply -f components.yaml
Filebeat Upgrade
In the configurations directory, change to the appropriate cluster context directory (bldr0-0, cabo0-0, tato0-1, and lnmt1-2) and run the following command:
$ kubectl apply -f filebeat-kubernetes.yamlUpdate File and Directory Permissions and Manifests
In the postinstall directory, run the update -s [site] script. This will perform the following steps.
- Add the cluster-name to the kube-controller-manager.yaml file
- Update the imagePullPolicy and image lines to all manifests
- Add the AlwaysPullImages and ResourceQuota admission controllers to the kube-apiserver.yaml file
- Update the permissions of all files and directories.