Upgrading Kubernetes Clusters
This document provides a guide to upgrading the Kubernetes clusters in the quickest manner. Much of the upgrade process can be done using Ansible Playbooks. There are a few processes that need to be done centrally on the tool server. And the OS and control plane updates are also manual in part due to the requirement to manually remove servers from the Kubernetes API pool.
In most cases, examples are not provided as it is assumed that you are familiar with the processes and can perform the updates without having to be reminded of how to verify.
For any process that is performed with an Ansible Playbook, it is assumed you are on the lnmt1cuomtool11 server in the /usr/local/admin/playbooks/cschelin/kubernetes directory. All Ansible related steps expect to start from that directory. In addition, the application of pod configurations will be in the configurations subdirectory.
Perform Upgrades
Patch Servers
Patch the control plane master servers one at a time and esure the cluster is healthy before continuing to the second and third master servers.
Drain each worker prior to patching and rebooting the worker node.
$ kubectl drain [nodename] --delete-local-data --ignore-daemonsetsPatch the server and reboot
yum upgrade -y
shutdown -t 0 now -rRejoin the worker node to the pool.
kubectl uncordon [nodename]Update Versionlock And Components
In the upgrade directory, run the update -t [tag] script. This will install yum-plugin-versionlock if missing, remove the old versionlocks, create new versionlocks for kubernetes, kubernetes-cni, and docker, and then the components will be upgraded.
Upgrade Kubernetes
Using the kubeadm command on the first master server, upgrade the first master server.
# kubeadm upgrade apply 1.18.8Upgrade Control Planes
On the second and third master, run the kubeadm upgrade apply 1.18.8 command and the control plane will be upgraded.
Update kube-proxy
Check the kube-proxy daemonset and update the image tag if required.
$ kubectl edit daemonset kube-proxy -n kube-system- Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000
Save the changes
Update coredns
Check the coredns-deployment and update the image tag if required.
$ kubectl edit deployment corednss -n kube-system- Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000
Save the changes.
Restart kubelet and docker
In the restart directory, run the update -t [tag] script. This will restart kubelet and docker on all servers.
Calico Upgrade
In the configurations/calico directory, run the following command:
$ kubectl apply -f calico.yamlcalicoctl Upgrade
Pull the updated calicoctl binary and copy it to /usr/local/bin.
$ curl -O -L https://github.com/projectcalico/calicoctl/releases/download/v3.16.0/calicoctlUpdate File and Directory Permissions and Manifests
In the postinstall directory, run the update -t [tag] script. This will perform the following steps.
- Add the cluster-name to the kube-controller-manager.yaml file
- Update the imagePullPolicy and image lines to all manifests
- Add the AlwaysPullImages and ResourceQuota admission controllers to the kube-apiserver.yaml file.
- Update the permissions of all files and directories.
Filebeat Upgrade
In the configurations directory, change to the appropriate cluster context directory, bldr0-0, cabo0-0, tato0-1, and lnmt1-2 and run the following command.
$ kubectl apply -f filebeat-kubernetes.yaml