Kubernetes Ansible Upgrade to 1.20.6

Upgrading Kubernetes Clusters

This document provides a guide to upgrading the Kubernetes clusters in the quickest manner. Much of the upgrade process can be done using Ansible Playbooks. There are a few processes that need to be done centrally on the tool server. And the OS and control plane updates are also manual in part due to the requirement to manually remove servers from the Kubernetes API pool.

In most cases, examples are not provided as it is assumed that you are familiar with the processes and can perform the updates without having to be reminded of how to verify.

For any process that is performed with an Ansible Playbook, it is assumed you are on the lnmt1cuomtool11 server in the /usr/local/admin/playbooks/cschelin/kubernetes directory. All Ansible related steps expect to start from that directory. In addition, the application of pod configurations will be in the configurations subdirectory.

Perform Upgrades

Patch Servers

In the 00-osupgrade directory, you’ll be running the master and worker scripts. I recommend opening two windows, one for master and one for worker, and running each script with master -t [tag] and worker -t [tag]. This will verify a node is Ready, drain the node from the pool if a worker, perform a yum upgrade and reboot, uncordon again if a worker, and verify the nodes are Ready again. Should a node fail to be ready in time, the script will exit.

Update Versionlock

In the 03-packages directory, run the update -t [tag] script. This will install yum-plugin-versionlock if missing, remove old versionlocks, create new versionlocks for kubernetes, kubernetes-cni, and docker, and then the components will be upgraded.

Upgrade Kubernetes

Using the kubeadm command on the first master server, upgrade the first master server.

# kubeadm upgrade apply v1.20.6
Update Control Planes

On the second and third master, run the kubeadm upgrade apply v1.20.6 command and the control plane will be upgraded.

Update kube-proxy

Check the kube-proxy daemonset and update the image tag if required.

kubectl edit daemonset kube-proxy -n kube-system
  • Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

Save the changes.

Update coredns

Check the coredns deployment and update the image tag if required.

kubectl edit deployment coredns -n kube-system
  • Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

Save the changes.

Restart kubelet and docker

In the 04-kubelet directory, run the update -t [tag] script. This will restart kubelet and docker on all servers.

Calico Upgrade

In the configurations/calico directory, run the following command:

$ kubectl apply -f calico.yaml

calicoctl Upgrade

Pull the updated calicoctl binary and copy it to /usr/local/bin. It’s likely already there but verify.

$ curl -O -L  https://github.com/projectcalico/calicoctl/releases/download/v3.18.2/calicoctl

kube-state-metrics Upgrade

In the configurations directory, /kube-state-metrics directory, run the following command:

$ kubectl apply -f kube-state-metrics/

metrics-server Upgrade

In the configurations/metrics-server directory, run the following command:

$ kubectl apply -f components.yaml

Filebeat Upgrade

In the configurations directory, change to the appropriate cluster context directory (bldr0-0, cabo0-0, tato0-1, and lnmt1-2) and run the following command:

$ kubectl apply -f filebeat-kubernetes.yaml

Update File and Directory Permissions and Manifests

In the postinstall directory, run the update -s [site] script. This will perform the following steps.

  • Add the cluster-name to the kube-controller-manager.yaml file
  • Update the imagePullPolicy and image lines to all manifests
  • Add the AlwaysPullImages and ResourceQuota admission controllers to the kube-apiserver.yaml file
  • Update the permissions of all files and directories.
This entry was posted in Computers, Kubernetes and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *