Kubernetes Preparation Steps for 1.20.6

Posted on March 24, 2021 by Freejack

Upgrading Kubernetes Clusters

The purpose of this document is to provide the background information on what is being upgraded, what versions, and the steps required to prepare for the upgrade itself. These steps are only done once. Once all these steps have been completed and all the configurations checked into github and gitlab, all clusters are then ready to be upgraded.

Reference links to product documentation at the end of this document.

Upgrade Preparation Steps

Upgrades to the Sandbox environment are done a few weeks before the official release for more in depth testing. Checking the release docs, changelog, and general operational status for the various tools that are in use.

Server Preparations

With the possibility of an upgrade to Spacewalk and to ensure the necessary software is installed prior to the upgrade, make sure all repositories are enabled and that the yum-plugin-versionlock software is installed.

Enable Repositories

Check the Spacewalk configuration and ensure that upgrades are coming from the local server and not from the internet.

Install yum versionlock

The critical components of Kubernetes are locked into place using the versionlock yum plugin. If not already installed, install it before beginning work.

# yum install yum-plugin-versionlock -y

Load Images

Next step is to load all the necessary Kubernetes, etcd, and additional images like coredns to the local repository so that all the clusters aren’t pulling all images from the internet. As a note, pause:3.1 has been upgraded to pause:3.2. Make sure you pull and update the image.

# docker pull k8s.gcr.io/kube-apiserver:v1.20.6
v1.20.6: Pulling from kube-apiserver
d94d38b8f0e6: Pull complete
6ee16ead6dee: Pull complete
ee5e6c27aaae: Pull complete
Digest: sha256:e6d960baa4219fa810ee26da8fe8a92a1cf9dae83b6ad8bda0e17ee159c68501
Status: Downloaded newer image for k8s.gcr.io/kube-apiserver:v1.20.6
k8s.gcr.io/kube-apiserver:v1.20.6
 
# docker pull k8s.gcr.io/kube-controller-manager:v1.20.6
v1.20.6: Pulling from kube-controller-manager
d94d38b8f0e6: Already exists
6ee16ead6dee: Already exists
a484c6338761: Pull complete
Digest: sha256:a1a6e8dbcf0294175df5f248503c8792b3770c53535670e44a7724718fc93e87
Status: Downloaded newer image for k8s.gcr.io/kube-controller-manager:v1.20.6
k8s.gcr.io/kube-controller-manager:v1.20.6
 
# docker pull k8s.gcr.io/kube-scheduler:v1.20.6
v1.20.6: Pulling from kube-scheduler
d94d38b8f0e6: Already exists
6ee16ead6dee: Already exists
1db6741b5f3c: Pull complete
Digest: sha256:ebb0350893fcfe7328140452f8a88ce682ec6f00337015a055d51b3fe0373429
Status: Downloaded newer image for k8s.gcr.io/kube-scheduler:v1.20.6
k8s.gcr.io/kube-scheduler:v1.20.6
 
# docker pull k8s.gcr.io/kube-proxy:v1.20.6
v1.20.6: Pulling from kube-proxy
e5a8c1ed6cf1: Pull complete
f275df365c13: Pull complete
6a2802bb94f4: Pull complete
cb3853c52da4: Pull complete
db342cbe4b1c: Pull complete
9a72dd095a53: Pull complete
a6a3a90a2713: Pull complete
Digest: sha256:7c1710c965f55bca8d06ebd8d5774ecd9ef924f33fb024e424c2b9b565f477dc
Status: Downloaded newer image for k8s.gcr.io/kube-proxy:v1.20.6
k8s.gcr.io/kube-proxy:v1.20.6
 
# docker pull k8s.gcr.io/pause:3.4.1
3.4.1: Pulling from pause
fac425775c9d: Pull complete
Digest: sha256:6c3835cab3980f11b83277305d0d736051c32b17606f5ec59f1dda67c9ba3810
Status: Downloaded newer image for k8s.gcr.io/pause:3.4.1
k8s.gcr.io/pause:3.4.1
 
# docker image ls
REPOSITORY                                                 TAG           IMAGE ID       CREATED         SIZE
k8s.gcr.io/kube-proxy                                      v1.20.6       9a1ebfd8124d   12 days ago     118MB
k8s.gcr.io/kube-scheduler                                  v1.20.6       b93ab2ec4475   12 days ago     47.2MB
k8s.gcr.io/kube-controller-manager                         v1.20.6       560dd11d4550   12 days ago     116MB
k8s.gcr.io/kube-apiserver                                  v1.20.6       b05d611c1af9   12 days ago     122MB
k8s.gcr.io/pause                                           3.4.1         0f8457a4c2ec   3 months ago    683kB

Next up is to tag all the images so they’ll be hosted locally on the bldr0cuomrepo1.internal.pri server.

# docker tag k8s.gcr.io/kube-apiserver:v1.20.6          bldr0cuomrepo1.internal.pri:5000/kube-apiserver:v1.20.6
# docker tag k8s.gcr.io/kube-controller-manager:v1.20.6 bldr0cuomrepo1.internal.pri:5000/kube-controller-manager:v1.20.6
# docker tag k8s.gcr.io/kube-scheduler:v1.20.6          bldr0cuomrepo1.internal.pri:5000/kube-scheduler:v1.20.6
# docker tag k8s.gcr.io/kube-proxy:v1.20.6              bldr0cuomrepo1.internal.pri:5000/kube-proxy:v1.20.6
# docker tag k8s.gcr.io/pause:3.4.1                     bldr0cuomrepo1.internal.pri:5000/pause:3.4.1
 
# docker image ls
REPOSITORY                                                 TAG           IMAGE ID       CREATED         SIZE
bldr0cuomrepo1.internal.pri:5000/kube-proxy                v1.20.6       9a1ebfd8124d   12 days ago     118MB
k8s.gcr.io/kube-proxy                                      v1.20.6       9a1ebfd8124d   12 days ago     118MB
bldr0cuomrepo1.internal.pri:5000/kube-controller-manager   v1.20.6       560dd11d4550   12 days ago     116MB
k8s.gcr.io/kube-controller-manager                         v1.20.6       560dd11d4550   12 days ago     116MB
k8s.gcr.io/kube-scheduler                                  v1.20.6       b93ab2ec4475   12 days ago     47.2MB
bldr0cuomrepo1.internal.pri:5000/kube-scheduler            v1.20.6       b93ab2ec4475   12 days ago     47.2MB
k8s.gcr.io/kube-apiserver                                  v1.20.6       b05d611c1af9   12 days ago     122MB
bldr0cuomrepo1.internal.pri:5000/kube-apiserver            v1.20.6       b05d611c1af9   12 days ago     122MB
bldr0cuomrepo1.internal.pri:5000/pause                     3.4.1         0f8457a4c2ec   3 months ago    683kB
k8s.gcr.io/pause                                           3.4.1         0f8457a4c2ec   3 months ago    683kB

The final step is to push them all up to the local repository.

# docker push bldr0cuomrepo1.internal.pri:5000/kube-apiserver:v1.20.6
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-apiserver]
d88bc16e0414: Pushed
a06ec64d2560: Pushed
28699c71935f: Pushed
v1.20.6: digest: sha256:d21627934fb7546255475a7ab4472ebc1ae7952cc7ee31509ee630376c3eea03 size: 949
 
# docker push bldr0cuomrepo1.internal.pri:5000/kube-controller-manager:v1.20.6
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-controller-manager]
1387661b583c: Pushed
a06ec64d2560: Mounted from kube-apiserver
28699c71935f: Mounted from kube-apiserver
v1.20.6: digest: sha256:ca13f2bf278e3157d75fd08a369390b98f976c6af502d4579a9ab62b97248b5b size: 949
 
# docker push bldr0cuomrepo1.internal.pri:5000/kube-scheduler:v1.20.6
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-scheduler]
f17938017a0a: Pushed
a06ec64d2560: Mounted from kube-controller-manager
28699c71935f: Mounted from kube-controller-manager
v1.20.6: digest: sha256:eee174e9eb4499f31bfb10d0350de87ea90431f949716cc4af1b5c899aab2058 size: 949
 
# docker push bldr0cuomrepo1.internal.pri:5000/kube-proxy:v1.20.6
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-proxy]
0c96004b5be1: Pushed
94812b0f02ce: Pushed
3a90582021f9: Pushed
f6be8a0f65af: Pushed
2b046f2c8708: Pushed
6ee930b14c6f: Pushed
f00bc8568f7b: Pushed
v1.20.6: digest: sha256:1689b5ac14d4d6e202a6752573818ce952e0bd3359b6210707b8b2031fedaa4d size: 1786
 
# docker push bldr0cuomrepo1.internal.pri:5000/pause:3.4.1
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/pause]
915e8870f7d1: Pushed
3.4.1: digest: sha256:9ec1e780f5c0196af7b28f135ffc0533eddcb0a54a0ba8b32943303ce76fe70d size: 526

Software Preparations

This section describes the updates that need to be made to the various containers that are installed in the Kubernetes clusters. Most of the changes involve updating the location to point to my Docker Repository vs pulling directly from the Internet.

You’ll need to clone if new, or pull the current playbook repo from gitlab as all the work will be done in various directories under the kubernetes/configurations directory. You’ll want to do that before continuing. All subsequent sections assume you’re in the kubernetes/configurations directory.

$ git clone git@lnmt1cuomgitlab.internal.pri:external-unix/playbooks.git
$ git pull git@lnmt1cuomgitlab.internal.pri:external-unix/playbooks.git

Make sure you add and commit the changes to your repo.

$ git add [file]
$ git commit [file] -m "commit comment"

And once done with all the updates, push the changes back up to gitlab.

$ git push

Update calico.yaml

In the calico directory run the following command to get the current calico.yaml file:

$ curl https://docs.projectcalico.org/manifests/calico.yaml -O

Basically grep out the image lines and pull the new images down to the local repository in order to retrieve the images locally.

# docker pull docker.io/calico/cni:v3.18.2
v3.18.2: Pulling from calico/cni
69606a78e084: Pull complete
85f85638f4b8: Pull complete
70ce15fa0c8a: Pull complete
Digest: sha256:664e1667fae09516a170ddd86e1a9c3bd021442f1e1c1fad19ce33d5b68bb58e
Status: Downloaded newer image for calico/cni:v3.18.2
docker.io/calico/cni:v3.18.2
 
# docker pull docker.io/calico/pod2daemon-flexvol:v3.18.2
v3.18.2: Pulling from calico/pod2daemon-flexvol
a5a0edbd6170: Pull complete
b10b71798d0d: Pull complete
5c3c4f282980: Pull complete
052e1842c6c3: Pull complete
6f392ce4dbcf: Pull complete
bc1f9a256ba0: Pull complete
fa4be31a19e9: Pull complete
Digest: sha256:7808a18ac025d3b154a9ddb7ca6439565d0af52a37e166cb1a14dcdb20caed67
Status: Downloaded newer image for calico/pod2daemon-flexvol:v3.18.2
docker.io/calico/pod2daemon-flexvol:v3.18.2
 
# docker pull docker.io/calico/node:v3.18.2
v3.18.2: Pulling from calico/node
2aee75817f4e: Pull complete
e1c64009c125: Pull complete
Digest: sha256:c598c6d5f43080f4696af03dd8784ad861b40c718ffbba5536b14dbf3b2349af
Status: Downloaded newer image for calico/node:v3.18.2
docker.io/calico/node:v3.18.2
 
# docker pull docker.io/calico/kube-controllers:v3.18.2
v3.18.2: Pulling from calico/kube-controllers
94ca07728981: Pull complete
c86a87d48320: Pull complete
f257a15e509c: Pull complete
8aad47abc588: Pull complete
Digest: sha256:ae544f188f2bd9d2fcd4b1f2b9a031c903ccaff8430737d6555833a81f4824d1
Status: Downloaded newer image for calico/kube-controllers:v3.18.2
docker.io/calico/kube-controllers:v3.18.2

Then tag the images for local storage.

# docker tag calico/cni:v3.18.2                bldr0cuomrepo1.internal.pri:5000/cni:v3.18.2
# docker tag calico/pod2daemon-flexvol:v3.18.2 bldr0cuomrepo1.internal.pri:5000/pod2daemon-flexvol:v3.18.2
# docker tag calico/node:v3.18.2               bldr0cuomrepo1.internal.pri:5000/node:v3.18.2
# docker tag calico/kube-controllers:v3.18.2   bldr0cuomrepo1.internal.pri:5000/kube-controllers:v3.18.2

Then push them up to the local repository.

# docker push bldr0cuomrepo1.internal.pri:5000/cni:v3.18.2
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/cni]
145c410196dc: Pushed
aec93328a278: Pushed
fd6f5b9d2ec9: Pushed
v3.18.2: digest: sha256:42ffea5056c9b61783423e16390869cdc16a8797eb9231cf7c747fe70371dfef size: 946
 
# docker push bldr0cuomrepo1.internal.pri:5000/pod2daemon-flexvol:v3.18.2
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/pod2daemon-flexvol]
125832445a60: Pushed
682e2fee7907: Pushed
12f496e83a60: Pushed
45acaaeabd00: Pushed
427dd33e9f20: Pushed
76ecd8aaf249: Pushed
63c82d5fed4a: Pushed
v3.18.2: digest: sha256:f243b72138e8e1d0e6399d000c03f38a052f54234f3d3b8a292f3c868a51ab07 size: 1788
 
# docker push bldr0cuomrepo1.internal.pri:5000/node:v3.18.2
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/node]
7c3bf8ac29b3: Pushed
534f69678b53: Pushed
v3.18.2: digest: sha256:d51436d6da50afc73d9de086aa03f7abd6938ecf2a838666a0e5ccb8dee25087 size: 737
 
# docker push bldr0cuomrepo1.internal.pri:5000/kube-controllers:v3.18.2
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-controllers]
5d1855397d0b: Pushed
4769d3354700: Pushed
4ea3707886e0: Pushed
054ba5c2f771: Pushed
v3.18.2: digest: sha256:d8d2c4a98bbdbfd19fe2e4cc9492552852a9d11628e338142b1d1268b51593ce size: 1155

Edit the file, search for image: and insert in front of the images, the image path:

bldr0cuomrepo1.internal.pri:5000

Make sure you follow the documentation to update calicoctl to 3.18.2 as well.

Update metrics-server

In the metrics-server directory, back up the existing components.yaml file and run the following command to get the current components.yaml file:

$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.1/components.yaml

Run a diff against the two files to see what might have changed. Then edit the file, search for image: and replace k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000.

Download the new image and save it locally.

# docker pull k8s.gcr.io/metrics-server/metrics-server:v0.4.3
v0.4.3: Pulling from metrics-server/metrics-server
5dea5ec2316d: Pull complete
ef7ee42a1880: Pull complete
Digest: sha256:eb6b6153494087bde59ceb14e68280f1fbdd17cfff2efc3a68e30a1adfa8807d
Status: Downloaded newer image for k8s.gcr.io/metrics-server/metrics-server:v0.4.3
k8s.gcr.io/metrics-server/metrics-server:v0.4.3

Tag the image.

# docker tag k8s.gcr.io/metrics-server/metrics-server:v0.4.3 bldr0cuomrepo1.internal.pri:5000/metrics-server:v0.4.3

And push the newly tagged image.

# docker push bldr0cuomrepo1.internal.pri:5000/metrics-server:v0.4.3
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/metrics-server]
abc161b95845: Pushed
417cb9b79ade: Pushed
v0.4.3: digest: sha256:2b6814cb0b058b753cb6cdfe906493a8128fabb03d405f60024a47ab49ddaa09 size: 739

Update kube-state-metrics

Updating kube-state-metrics is a bit more involved as there are several files that are part of the distribution however you only need a small subset. You’ll need to clone or pull the kube-state-metrics repo.

$ git clone https://github.com/kubernetes/kube-state-metrics.git

Once you have the repo, in the kube-state-metrics/examples/standard directory, copy all the files into the playbooks kube-state-metrics directory.

Edit the deployment.yaml file, search for image: and replace quay.io with bldr0cuomrepo1.internal.pri:5000

After you’ve updated the files, download the image:

# docker pull k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0
v2.0.0: Pulling from kube-state-metrics/kube-state-metrics
5dea5ec2316d: Already exists
2c0aab77c223: Pull complete
Digest: sha256:eb2f41024a583e8795213726099c6f9432f2d64ab3754cc8ab8d00bdbc328910
Status: Downloaded newer image for k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0
k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0

Tag the image.

# docker tag k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0 bldr0cuomrepo1.internal.pri:5000/kube-state-metrics:v2.0.0

And push the newly tagged image.

# docker push bldr0cuomrepo1.internal.pri:5000/kube-state-metrics:v2.0.0
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/kube-state-metrics]
d2bc11882435: Pushed
417cb9b79ade: Mounted from metrics-server
v2.0.0: digest: sha256:ee13833414a49b0d2370e8edff5844eba96630cda80cfcd37c444bf88522cc51 size: 738

Update filebeat-kubernetes.yaml

In the filebeat directory, run the following command to get the current filebeat-kubernetes.yaml file:

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.12/deploy/kubernetes/filebeat-kubernetes.yaml

Change all references in the filebeat-kubernetes.yaml file from kube-system to monitoring. If a new installation, create the monitoring namespace.

Update the local repository with the new docker image.

# docker pull docker.elastic.co/beats/filebeat:7.12.1
7.12.1: Pulling from beats/filebeat
a4f595742a5b: Pull complete
f7bc9401458a: Pull complete
ce7f9b59a9d3: Pull complete
e0ba09632c1a: Pull complete
3a0a0a9a5b5f: Pull complete
4f7abff72235: Pull complete
8cf479d85574: Pull complete
3b62c2ebd4b6: Pull complete
79a6ebf558dc: Pull complete
0c22790a6b07: Pull complete
dfd98a660972: Pull complete
Digest: sha256:e9558ca6e2df72a7933d4f175d85e8cf352da08bc32d97943bb844745d4a063a
Status: Downloaded newer image for docker.elastic.co/beats/filebeat:7.12.1
docker.elastic.co/beats/filebeat:7.12.1

Tag the image appropriately.

# docker tag docker.elastic.co/beats/filebeat:7.12.1 bldr0cuomrepo1.internal.pri:5000/filebeat:7.12.1

Finally, push it up to the local repository.

# docker push bldr0cuomrepo1.internal.pri:5000/filebeat:7.12.1
The push refers to repository [bldr0cuomrepo1.internal.pri:5000/filebeat]
446d15d628e2: Pushed
19bc11b9258e: Pushed
8ee55e79c98f: Pushed
851de8b3f92f: Pushed
eacdcb47588f: Pushed
bc27d098296e: Pushed
9c4f2da5ee8b: Pushed
2c278752a013: Pushed
bd82c7b8fd60: Pushed
f9b1f5eda8ab: Pushed
174f56854903: Pushed
7.12.1: digest: sha256:02a034166c71785f5c2d1787cc607994f68aa0521734d11da91f8fbd0cfdc640 size: 2616

Once the image is hosted locally, copy the file into each of the cluster directories and make the following changes.

DaemonSet Changes

In the filebeat folder are two files. A config file and an update file. These files automatically make changes to the filebeat-kubernetes.yaml file based on some of the changes that are performed below. The below changes are made to prepare for the script which populates the different clusters with correct information.

  • Switches the docker.elastic.co/beats image with bldr0cuomrepo1.internal.pri:5000
  • Replaces <elasticsearch> with the actual ELK Master server name
  • Switches the kube-system namespace with monitoring. You’ll need to ensure the monitoring namespace has been created before applying this .yaml file.
  • Replaces DEPLOY_ENV with the expected deployment environment name; dev, sqa, staging, or prod. These names are used in the ELK cluster to easily identify where the logs are sourced.

In order for the script to work, change the values in the following lines to match:

        - name: ELASTICSEARCH_HOST
          value: "<elasticsearch>"
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: ""
        - name: ELASTICSEARCH_PASSWORD
          value: ""

In addition, remove the following lines. They confuse the container if they exist.

        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:

Add the default username and password to the following lines as noted:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME:elastic}
      password: ${ELASTICSEARCH_PASSWORD:changeme}
ConfigMap Changes

In the ConfigMap section, activate the filebeat.autodiscover section by uncommenting it and delete the filebeat.inputs configuration section. In the filebeat.autodiscover section make the following three changes:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      host: ${NODE_NAME}                          # rename node to host
      hints.enabled: true
      hints.default_config.enabled: false         # add this line
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
        exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines  # add this line

In the processors section, remove the cloud.id and cloud.auth lines, add the following lines, and change DEPLOY_ENV to the environment filebeat is being deployed to: dev, sqa, staging, or prod. Indentation is important!

processors:
- add_cloud_metadata:
- add_host_metadata:
- add_fields:                             # add these 4 lines. pay attention to indentation!
target: ''
fields:
environment: 'DEPLOY_ENV' 
Elastic Stack in Development

This Elastic Stack cluster is used by the Development Kubernetes clusters. Update the files in the bldr0-0 directory.

- name: ELASTICSEARCH_HOST
  value: bldr0cuomifem1.internal.pri
Elastic Stack in QA

This Elastic Stack cluster is used by the QA Kubernetes clusters. Update the files in the cabo0-0 directory.

- name: ELASTICSEARCH_HOST
  value: cabo0cuomifem1.internal.pri
Elastic Stack in Staging

This Elastic Stack cluster is used by the Staging Kubernetes clusters. Update the files in the tato0-1 directory.

- name: ELASTICSEARCH_HOST
  value: tato0cuomifem1.internal.pri
Elastic Stack in Production

This Elastic Stack cluster is used by the Production Kubernetes cluster. Update the file in the lnmt1-2 directory.

- name: ELASTICSEARCH_HOST
  value: lnmt1cuelkmstr1.internal.pri
This entry was posted in Computers, Kubernetes and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *