Preparing Kubernetes

Posted on November 20, 2022 by Freejack

Overview

This article will provide a howto on preparing hosts to install Kubernetes 1.25.7 on CentOS 7 using kubeadm. I’ll be using CRI-O as the container environment and Calico for the network layer. A followup article will provide instructions in building the cluster and post installation needs.

Note that I tried Rocky Linux 8 but podman isn’t current enough for CRI-O and is throwing errors due to a change in the configuration file from a single entry to multiple entries.

Insecure Registries

Currently I’m using an on-prem insecure registry. I installed the docker distribution software which works well enough to host local images. Then on a docker server, I pull the necessary images, tag them with the local information, and then push them to the new local registry. Then I update kubernetes manifests and other tools to point to the local registry. With this, I’m not pulling images from the internet every time I make some change or another.

Prepare Hosts

There are a few things that need to be done with the hosts to make them ready.

Container Runtime

In order to use a container run time, you’ll need to create a couple of files. You’ll be creating a bridge and overlay file and modify the system with sysctl.

First time in /etc/modules-load-d create a br_netfilter.conf file.

br_netfilter

Next create the /etc/modules-load-d overlay.conf file.

br_netfilter

You can either restart the system or simply use modprobe to load the modules.

modprobe overlay
modprobe br_netfilter

Next create /etc/sysctl.d/kubernetes.conf and add the following lines:

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

Again, restart the system or simply reload the sysctl table:

sysctl --system

Disable swap

First off is to disable and remove swap from the all the nodes, control and worker. Since it’s Kubernetes and it manages resources, swap is not needed.

  • Remove the /dev/mapper/vg00-swap line from /etc/fstab
  • Remove rd.lvm.lv=vg00/swap from /etc/default/grub and run grub2-mkconconfig -o /boot/grub2/grub.cfg to rebuild the grub.cfg file.
  • Disable swap by running swapoff -v /dev/mapper/vg00-swap
  • Run the umount /dev/mapper/vg00-swap command to remove swap the run lvremove /dev/mapper/vg00-swap to recover the space.

If SELinux is configured, ensure the SELINUX line is set to permissive in /etc/selinux/config. You’ll need to reboot of course to enable this.

You may want to do some Quality of Service management. If so, install the iproute-tc tool. See the References section for further information on the software.

Firewalls

I have firewalls running around on my servers as I follow the zero-trust networking model, however because I’m using Calico for my network layer, it handles it for me so you need to disable the firewall on all nodes.

Docker

At least for CentOS 7 you’ll install the docker and tools on all nodes.

yum install -y docker docker-common docker-client

Configure docker to allow access to the on-prem insecure registries. Without this, docker will not pull the images. In addition, you want to use journald for logging. Update the /etc/docker/daemon.json file as follows:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "5"
  },
  "insecure-registries": ["bldr0cuomrepo1.dev.internal.pri:5000"]
}

In addition, update the docker system startup file and add the following flag.

--log-driver=journald

Container Runtime

Now the systems are ready for CRI-O. You’ll need to add a couple of repositories to your control nodes before doing the installation plus as of 1.24.0, you’ll have the option of selecting a CNI plugin. I’ll be using the containernetworking-plugins as that’s now it was set up but you have the option to select a different one if you like.

Configure Repositories

You’ll need to add the two repositories as provided below. While we can pull the files from the CRI-O website, as always we want consistency across the clusters. We are installing 1.24.0 on CentOS 7.

First the crio.repo file. Save it in /etc/yum.repos.d/crio.repo

[devel_kubic_libcontainers_stable_cri-o_1.24]
name=devel:kubic:libcontainers:stable:cri-o:1.24 (CentOS_7)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.24/CentOS_7/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.24/CentOS_7/repodata/repomd.xml.key
enabled=1

Next is the stable.repo. Again save it in /etc/yum.repos.d/stable.repo

[devel_kubic_libcontainers_stable]
name=Stable Releases of Upstream github.com/containers packages (CentOS_7)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_7/repodata/repomd.xml.key
enabled=1

Install the crio package.

yum install crio

Then install the CNI of choice.

yum install containernetworking-plugins

In order for CRI-O to know about the on-prem insecure registries, you’ll need to update the /etc/containers/registries.conf. Add the following TOML formatted block of code.

[[registry]]
prefix = "bldr0cuomrepo1.dev.internal.pri:5000"
insecure = true
location = "bldr0cuomrepo1.dev.internal.pri:5000"

The pause container isn’t displayed when getting a listing of pods, but it’s used by Kubernetes to manage the network namespace so restarting or crashing pods don’t lose their network configuration. In order to point to a local insecure registry, you have to update the /etc/crio/crio.conf file with the following line:

pause_image = "bldr0cuomrepo1.dev.internal.pri:5000/pause:3.6"

When all are installed. Enable and start crio.

systemctl enable crio
systemctl start crio

Kubernetes Binaries

In order to install kubernetes binaries, you’ll first need to install the kubernetes repository into /etc/yum.repos.d. Create the file, kubernetes.repo and add the following lines.

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl

And now, install the necessary binaries.

dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Next enable kubelet. You won’t be able to start it as the config.yaml file doesn’t exist yet. That’s created when you run kubeadm.

systemctl enable kubelet

Build kubeadm Config

There are multiple options for the kubeadm-config.yaml file. Here is the one I’m using when building the cluster. This file should only be on the first control node as once the cluster is started, you’ll have commands to run to join other control and worker nodes to the first control node.

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  imagePullPolicy: Always
---
apiVersion: kubeadm.k8s.io/v1beta3
clusterName: "bldr"
controlPlaneEndpoint: "bldr0cuomvip1.dev.internal.pri:6443"
etcd:
  local:
    imageRepository: "bldr0cuomrepo1.dev.internal.pri:5000"
imageRepository: "bldr0cuomrepo1.dev.internal.pri:5000"
kind: ClusterConfiguration
kubernetesVersion: "1.25.7"
networking:
  podSubnet: "10.42.0.0/16"
  serviceSubnet: "10.69.0.0/16"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

There are three sections here to detail.

InitConfiguration

For security purposes, we want to adjust all images to be pulled every time it needs to be loaded. Since the image repository is on-prem, having it be set to Always isn’t a big issue.

ClusterConfiguration

There are several options we’ll set to make sure we are running properly when initializing the cluster.

clusterName: I have four environments that will have clusters. The sites are bldr (dev), cabo (qa), tato (stage), and lnmt (production). Set this to one of the environments.

controlPlaneEndpoint: This is the HAProxy VIP along with the port of 6443.

imageRepository: This is the local image repository, in this case bldr0cuomrepo1.dev.internal.pri:5000. It’s set for the etcd binary and the three kubernetes binaries.

kubernetesVersion: Set it to the version being installed, in this case 1.25.7.

networking.podSubnet: Set to the network all the pods will be started on.

networking.serviceSubnet: Set to the network all internal services will use.

KubeletConfiguration

This is used by the metrics-server in order to access the cluster and return statistics. This setting will be applied to every servers kubelet config.yaml file plus to the cluster kubeadm-config configmap.

As a note, Certificate Signing Requests (CSRs) will need to be approved once the cluster is up.

Conclusion

The servers are all prepared and ready to be started. Log in to the first control node and follow the instructions for building the cluster.

References

This entry was posted in Computers, Kubernetes and tagged . Bookmark the permalink.

One Response to Preparing Kubernetes

  1. Pingback: Kubernetes Index | Motorcycle Touring

Leave a Reply

Your email address will not be published. Required fields are marked *