Upgrading Kubernetes Clusters
The purpose of the document is to provide the background information on what is being upgraded, what versions, and the steps required to prepare for the upgrade itself. These steps are only done once. Once all these steps have been completed and all the configurations checked into gitlab, all clusters are then ready to be upgraded.
Upgrade Preparation Steps
Upgrades to the sandbox environment are done a few weeks before the official release for more in depth testing. Checking the release docs, changelog, and general operational status for the various tools that are in use.
Sever Preparations
With the possibility of an upgrade to Spacewalk and to ensure the necessary software is installed prior to the upgrade, make sure all repositories are enabled and that the yum-plugin-versionlock software is installed.
Enable Repositories
Check the Spacewalk configuration and ensure that upgrades are coming from the local server and not from the internet.
Install yum versionlock
The critical components of Kubernetes are locked into place using the versionlock yum plugin. If not already installed, install it before beginning work.
# yum install yum-plugin-versionlock -ySoftware Preparations
This section describes the updates that need to be made to the various containers that are installed in the Kubernetes clusters. Most of the changes involve updating the location to point to the local Docker repository vs pulling directly from the internet.
Ansible Playbooks
This section isn’t going to be instructions on setting up or using Ansible Playbooks. The updates to the various configurations are also saved with the Ansible Playbooks repo. You’ll make the appropriate changes to the updated configuration files and then push them back up to the gitlab server.
Update calico.yaml
In the calico directory, run the following command to get the current calico.yaml file.
$ curl https://docs.projectcalico.org/manifests/calico.yaml -OEdit the file, search for image: and insert in front of calico, the path to the local repository.
bldr0cuomrepo1.internal.pri:5000/Make sure you follow the documentation to update calicoctl to 3.16.0.
Update metrics-server
In the metrics-server directory, run the following command to get the current components.yaml file:
$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yamlEdit the file, search for image: and replace k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000/
Update kube-state-metrics
Updating kube-state-metrics is a bit more involved as there are several files that are part of the distribution, however you only need a small subset. You’ll need to clone or if you already have it, pull the kube-state-metrics repo.
$ git clone https://github.com/kubernetes/kube-state-metrics.gitOnce you have the repo, in the kube-state-metrics/examples/standard directory, copy all the files into the playbooks kube-state-metrics directory.
Edit the deployment.yaml file and replace quay.io with bldr0cuomrepo1.internal.pri:5000/
Update filebeat-kubernetes.yaml
In the filebeat directory, run the following command to get the current filebeat-kubernetes.yaml file:
$ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.9/deploy/kubernetes/filebeat-kubernetes.yamlChange all references in the filebeat-kubernetes.yaml file from kube-system to monitoring. If a new installation, create the monitoring namespace.
Then copy the file into each of the cluster directories and make the following changes.
DaemonSet Changes
In the DaemonSet section, replace the image location docker.elastic.co/filebeat:7.9.2 with bldr0cuomrepo1.internal.pri:5000/beats/filebeat:7.9.2. This pulls the image from our local repository vs from the Internet.
In order for the search and replace script to work the best, make the following changes:
- name: ELASTICSEARCH_HOST
value: "<elasticsearch>"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: ""
- name: ELASTICSEARCH_PASSWORD
value: ""In addition, remove the following lines. They confuse the container if they exist.
- name: ELASTIC_CLOUD_ID
value:
- name: ELASTIC_CLOUD_AUTH
value:Add the default username and password to the following lines as noted:
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME:elastic}
password: ${ELASTICSEARCH_PASSWORD:changeme}ConfigMap Changes
In the ConfigMap section, activate the filebeat.autodiscover section by uncommenting it and delete the filebeat.inputs configuration section. In the filebeat.autodiscover section, make the following three changes as noted with comments.
filebeat.autodiscover:
providers:
- type: kubernetes
host: ${NODE_NAME} # rename node to host
hints.enabled: true
hints.default_config.enabled: false # add this line
hints.default_config:
type: container
paths:
- /var/log/containers/*${data.kubernetes.container.id}.log
exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines # add this lineIn the processors section, remove the cloud.id and cloud.auth lines, add the following uncommented lines, and change DEPLOY_ENV to the environment filebeat is being deployed to: dev, sqa, staging, or prod.
# Add deployment environment field to every event to make it easier to sort between Dev and SQA logs.
# DEPLOY_ENV values: dev, sqa, staging, or prod
- add_fields:
target: ''
fields:
environment: 'DEPLOY_ENV'Elastic Stack in Dev and QA
This Elastic Stack cluster is used by the Development and QA Kubernetes clusters. Update the files in the bldr0-0 and cabo0-0 subdirectories.
- name: ELASTICSEARCH_HOST
value: bldr0cuomemstr1.internal.priElastic Stack in Staging
This Elastic Stack cluster is used by the Staging Kubernetes cluster. Update the files in the tato0-1 subdirectory.
- name: ELASTICSEARCH_HOST
value: tato0cuomelkmstr1.internal.priElastic Stack in Production
This Elastic Stack cluster is used by the Production Kubernetes Cluster. Update the file in the lnmt1-2 subdirectory.
- name: ELASTICSEARCH_HOST
value: lnmt1cuomelkmstr1.internal.pri