Kubernetes Manual Upgrade to 1.20.6

Upgrading Kubernetes Clusters

This documentation is intended to provide the manual process for upgrading the server Operating Systems, Kubernetes to 1.20.6, and any additional updates. This provides example output and should help in troubleshooting should the automated processes experience a problem.

All of the steps required to prepare for an installation should be completed prior to starting this process.

Server and Kubernetes Upgrades

Patch Servers

As part of quarterly upgrades, the Operating Systems for all servers need to be upgraded.

For the control plane, there isn’t a “pool” so just patch each server and reboot it. Do one server at a time and check the status of the cluster before moving to subsequent master servers on the control plane.

For the worker nodes, you’ll need to drain each of the workers before patching and rebooting. Run the following command to both confirm the current version of 1.19.6 and that all nodes are in a Ready state to be patched:

$ kubectl get nodes NAME                           STATUS   ROLES    AGE    VERSION bldr0cuomknode1.internal.pri   Ready    <none>   214d   v1.19.6 bldr0cuomknode2.internal.pri   Ready    <none>   214d   v1.19.6 bldr0cuomknode3.internal.pri   Ready    <none>   214d   v1.19.6 bldr0cuomkube1.internal.pri    Ready    master   214d   v1.19.6 bldr0cuomkube2.internal.pri    Ready    master   214d   v1.19.6 bldr0cuomkube3.internal.pri    Ready    master   214d   v1.19.6

To drain a server, patch, and then return the server to the pool, follow the steps below.

kubectl drain [nodename] --delete-local-data --ignore-daemonsets

Then patch the server and reboot:

yum upgrade -yshutdown -t 0 now -r

Finally bring the node back into the pool.

kubectl uncordon [nodename]

Update Versionlock Information

Currently the clusters have locked kubernetes to version 1.19.6, kubernetes-cni to version 0.8.7, and docker to 1.13.1-203. The locks on each server need to be removed and new locks put in place for the new versions of kubernetes, kubernetes-cni, and docker where appropriate.

Versionlock file location: /etc/yum/pluginconf.d/

Simply delete the existing locks:

/usr/bin/yum versionlock delete "kubelet.*" /usr/bin/yum versionlock delete "kubectl.*" /usr/bin/yum versionlock delete "kubeadm.*" /usr/bin/yum versionlock delete "kubernetes-cni.*" /usr/bin/yum versionlock delete "docker.*" /usr/bin/yum versionlock delete "docker-common.*" /usr/bin/yum versionlock delete "docker-client.*" /usr/bin/yum versionlock delete "docker-rhel-push-plugin.*"

And then add in the new locks at the desired levels:

/usr/bin/yum versionlock add "kubelet-1.20.6-0.*" /usr/bin/yum versionlock add "kubectl-1.20.6-0.*" /usr/bin/yum versionlock add "kubeadm-1.20.6-0.*" /usr/bin/yum versionlock "docker-1.13.1-204.*" /usr/bin/yum versionlock "docker-common-1.13.1-204.*" /usr/bin/yum versionlock "docker-client-1.13.1-204.*" /usr/bin/yum versionlock "docker-rhel-push-plugin-1.13.1-204.*"

Then install the updated kubernetes and docker binaries. Note that the versionlocked versions and the installed version must match:

/usr/bin/yum install kubelet-1.20.6-0.x86_64 /usr/bin/yum install kubectl-1.20.6-0.x86_64 /usr/bin/yum install kubeadm-1.20.6-0.x86_64 /usr/bin/yum install docker-1.13.1-204.git0be3e21.el7_8.x86_64 /usr/bin/yum install docker-common-1.13.1-204.git0be3e21.el7* /usr/bin/yum install docker-client-1.13.1-204.git0be3e21.el7* /usr/bin/yum install docker-rhel-push-plugin-1.13.1-204.git0be3e21.el7*

2.3 Upgrade Kubernetes

Using the kubeadm command on the first master server, you can review the plan and then upgrade the cluster:

# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.19.6 [upgrade/versions] kubeadm version: v1.20.6 I0427 17:46:38.139615   20479 version.go:254] remote version is much newer: v1.21.0; falling back to: stable-1.20 [upgrade/versions] Latest stable version: v1.20.6 [upgrade/versions] Latest stable version: v1.20.6 [upgrade/versions] Latest version in the v1.19 series: v1.19.10 [upgrade/versions] Latest version in the v1.19 series: v1.19.10   Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT   CURRENT       AVAILABLE kubelet     6 x v1.19.6   v1.19.10   Upgrade to the latest version in the v1.19 series:   COMPONENT                 CURRENT    AVAILABLE kube-apiserver            v1.19.6    v1.19.10 kube-controller-manager   v1.19.6    v1.19.10 kube-scheduler            v1.19.6    v1.19.10 kube-proxy                v1.19.6    v1.19.10 CoreDNS                   1.7.0      1.7.0 etcd                      3.4.13-0   3.4.13-0   You can now apply the upgrade by executing the following command:   kubeadm upgrade apply v1.19.10   _____________________________________________________________________   Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT   CURRENT       AVAILABLE kubelet     6 x v1.19.6   v1.20.6   Upgrade to the latest stable version:   COMPONENT                 CURRENT    AVAILABLE kube-apiserver            v1.19.6    v1.20.6 kube-controller-manager   v1.19.6    v1.20.6 kube-scheduler            v1.19.6    v1.20.6 kube-proxy                v1.19.6    v1.20.6 CoreDNS                   1.7.0      1.7.0 etcd                      3.4.13-0   3.4.13-0   You can now apply the upgrade by executing the following command:   kubeadm upgrade apply v1.20.6   _____________________________________________________________________     The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column.   API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no kubelet.config.k8s.io     v1beta1           v1beta1             no _____________________________________________________________________

There are likely newer versions of Kubernetes control plane containers available. In order to maintain consistency across all clusters, only upgrade the masters to 1.19.6:

# kubeadm upgrade apply v1.20.6 [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.20.6" [upgrade/versions] Cluster version: v1.19.6 [upgrade/versions] kubeadm version: v1.20.6 [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull' [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.20.6"... Static pod: kube-apiserver-bldr0cuomkube1.internal.pri hash: 2742aa8dcdc3cb47ed265f67f1a04783 Static pod: kube-controller-manager-bldr0cuomkube1.internal.pri hash: dd7adc86b875b67ba03820b12d904fa9 Static pod: kube-scheduler-bldr0cuomkube1.internal.pri hash: 6a43bc71ab534486758c1d56bd907ea3 [upgrade/etcd] Upgrading to TLS for etcd Static pod: etcd-bldr0cuomkube1.internal.pri hash: 7e320baf6cd06f441f462de7da1d6f05 [upgrade/staticpods] Preparing for "etcd" upgrade [upgrade/staticpods] Renewing etcd-server certificate [upgrade/staticpods] Renewing etcd-peer certificate [upgrade/staticpods] Renewing etcd-healthcheck-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-04-27-23-31-35/etcd.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: etcd-bldr0cuomkube1.internal.pri hash: 7e320baf6cd06f441f462de7da1d6f05 Static pod: etcd-bldr0cuomkube1.internal.pri hash: 7e320baf6cd06f441f462de7da1d6f05 Static pod: etcd-bldr0cuomkube1.internal.pri hash: 7e320baf6cd06f441f462de7da1d6f05 Static pod: etcd-bldr0cuomkube1.internal.pri hash: 7e320baf6cd06f441f462de7da1d6f05 ... [apiclient] Found 3 Pods for label selector component=etcd [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests040252515" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Renewing apiserver-etcd-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-04-27-23-31-35/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-bldr0cuomkube1.internal.pri hash: 2742aa8dcdc3cb47ed265f67f1a04783 Static pod: kube-apiserver-bldr0cuomkube1.internal.pri hash: 7426ddce1aafd033ae049eefb6d56b1e [apiclient] Found 3 Pods for label selector component=kube-apiserver [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade [upgrade/staticpods] Renewing controller-manager.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-04-27-23-31-35/kube-controller-manager.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-controller-manager-bldr0cuomkube1.internal.pri hash: dd7adc86b875b67ba03820b12d904fa9 Static pod: kube-controller-manager-bldr0cuomkube1.internal.pri hash: 281525a644d92747499c625139b84436 [apiclient] Found 3 Pods for label selector component=kube-controller-manager [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Preparing for "kube-scheduler" upgrade [upgrade/staticpods] Renewing scheduler.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2021-04-27-23-31-35/kube-scheduler.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-scheduler-bldr0cuomkube1.internal.pri hash: 6a43bc71ab534486758c1d56bd907ea3 Static pod: kube-scheduler-bldr0cuomkube1.internal.pri hash: aa70347866b81f5866423fcccb0c6aca [apiclient] Found 3 Pods for label selector component=kube-scheduler [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upgrade/postupgrade] Applying label node-role.kubernetes.io/control-plane='' to Nodes with label node-role.kubernetes.io/master='' (deprecated) [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.20" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy   [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.20.6". Enjoy!   [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.

Update Control Planes

On the second and third master, run the kubeadm upgrade apply v1.20.6 command and the control plane will be upgraded.

Update File and Directory Permissions

Verify the permissions match the table below once the upgrade is complete:

/etc/kubernetes/manifests/etcd.yaml	root:root	0644
/etc/kubernetes/manifests/kube-apiserver.yaml	root:root	0644
/etc/kubernetes/manifests/kube-controller-manager.yaml	root:root	0644
/etc/kubernetes/manifests/kube-scheduler	root:root	0644
/var/lib/etcd	root:root	0700
/etc/kubernetes/admin.conf	root:root	0644
/etc/kubernetes/scheduler.conf	root:root	0644
/etc/kubernetes/controller-manager.conf	root:root	0644
/etc/kubernetes/pki	root:root	0755
/etc/kubernetes/pki/ca.crt	root:root	0644
/etc/kubernetes/pki/apiserver.crt	root:root	0644
/etc/kubernetes/pki/apiserver-kubelet-client.crt	root:root	0644
/etc/kubernetes/pki/front-proxy-ca.crt	root:root	0644
/etc/kubernetes/pki/front-proxy-client.crt	root:root	0644
/etc/kubernetes/pki/sa.pub	root:root	0644
/etc/kubernetes/pki/ca.key	root:root	0600
/etc/kubernetes/pki/apiserver.key	root:root	0600
/etc/kubernetes/pki/apiserver-kubelet-client.key	root:root	0600
/etc/kubernetes/pki/front-proxy-ca.key	root:root	0600
/etc/kubernetes/pki/front-proxy-client.key	root:root	0600
/etc/kubernetes/pki/sa.key	root:root	0600
/etc/kubernetes/pki/etcd	root:root	0755
/etc/kubernetes/pki/etcd/ca.crt	root:root	0644
/etc/kubernetes/pki/etcd/server.crt	root:root	0644
/etc/kubernetes/pki/etcd/peer.crt	root:root	0644
/etc/kubernetes/pki/etcd/healthcheck-client.crt	root:root	0644
/etc/kubernetes/pki/etcd/ca.key	root:root	0600
/etc/kubernetes/pki/etcd/server.key	root:root	0600
/etc/kubernetes/pki/etcd/peer.key	root:root	0600
/etc/kubernetes/pki/etcd/healthcheck-client.key	root:root	0600

Update Manifests

During the kubeadm upgrade, the current control plane manifests are moved from /etc/kubernetes/manifests into /etc/kubernetes/tmp and new manifest files deployed. There are multiple settings and permissions that need to be reviewed and updated before the task is considered completed.

The kubeadm-config configmap has been updated to point to bldr0cuomrepo1.internal.pri:5000 however it and the various container configurations should be checked anyway. One of the issues is if it’s not updated or used, you’ll have to make the update manually including manually editing the kube-proxy daemonset configuration.

Note that when a manifest is updated, the associated image is reloaded. No need to manage the pods once manifests are updated.

etcd Manifest

Verify and update etcd.yaml

• Change imagePullPolicy to Always.
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

kube-apiserver Manifest

Verify and update kube-apiserver.yaml

• Add AlwaysPullImages and ResourceQuota admission controllers to the –enable-admission-plugins line
• Change imagePullPolicy to Always.
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

kube-controller-manager Manifest

Verify and update kube-controller-manager.yaml

• Add “ – –cluster-name=kubecluster-[site]” after “ – –cluster-cidr=192.168.0.0/16”
• Change imagePullPolicy to Always
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

kube-scheduler Manifest

Varify and update kube-scheduler.yaml

• Change imagePullPolicy to Always.
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

Update kube-proxy

Verify where the kube-proxy images is being loaded from. If not the local repository, you’ll need to edit the kube-proxy daemonset to change the imagePullPolicy. Check the image tag at the same time.

kubectl edit daemonset kube-proxy -n kube-system

• Change imagePullPolicy to Always
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

Save the changes.

Update coredns

Verify where the coredns images is being loaded from. If not the local repository, you’ll need to edit the coredns deployment to change the imagePullPolicy. Check the image tag at the same time.

kubectl edit deployment coredns -n kube-system

• Change imagePullPolicy to Always
• Change image switching k8s.gcr.io with bldr0cuomrepo1.internal.pri:5000

Save the changes.

Restart kubelet

Once done, kubelet and docker needs to be restarted on all nodes.

systemctl daemon-reload
systemctl restart kubelet
systemctl restart docker

Verify

Once kubelet has been restarted on all nodes, verify all nodes are at 1.20.6.

$ kubectl get nodes NAME                           STATUS   ROLES                  AGE    VERSION bldr0cuomknode1.internal.pri   Ready    <none>                 215d   v1.20.6 bldr0cuomknode2.internal.pri   Ready    <none>                 215d   v1.20.6 bldr0cuomknode3.internal.pri   Ready    <none>                 215d   v1.20.6 bldr0cuomkube1.internal.pri    Ready    control-plane,master   215d   v1.20.6 bldr0cuomkube2.internal.pri    Ready    control-plane,master   215d   v1.20.6 bldr0cuomkube3.internal.pri    Ready    control-plane,master   215d   v1.20.6

Configuration Upgrades

Configuration files are on the tool servers (lnmt1cuomtool11) in the /usr/local/admin/playbooks/cschelin/kubernetes/configurations directory and the expectation is you’ll be in that directory when directed to apply configurations.

Calico Upgrade

In the calico directory, run the following command:

$ kubectl apply -f calico.yaml configmap/calico-config unchanged customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured clusterrole.rbac.authorization.k8s.io/calico-kube-controllers configured clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged clusterrole.rbac.authorization.k8s.io/calico-node unchanged clusterrolebinding.rbac.authorization.k8s.io/calico-node unchanged daemonset.apps/calico-node configured serviceaccount/calico-node unchanged deployment.apps/calico-kube-controllers configured serviceaccount/calico-kube-controllers unchanged poddisruptionbudget.policy/calico-kube-controllers unchanged

After calico.yaml is applied, the calico-kube-controllers pod will restart and then the calico-node pod restarts to retrieve the updated image.

Pull the calicoctl binary and copy it to /usr/local/bin, then verify the version. Note that this has likely already been done on the tool server. Verify it before pulling the binary.

$ curl -O -L  https://github.com/projectcalico/calicoctl/releases/download/v3.18.2/calicoctl % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed 100   615  100   615    0     0    974      0 --:--:-- --:--:-- --:--:--   974 100 38.1M  100 38.1M    0     0  1505k      0  0:00:25  0:00:25 --:--:-- 1562k

Verification

$ calicoctl version Client Version:    v3.18.2 Git commit:        528c5860 Cluster Version:   v3.18.2 Cluster Type:      k8s,bgp,kubeadm,kdd

Update CNI File Permissions

Verify the permissions of the files once the upgrade is complete.

/etc/cni/net.d/10-calico-conflist	root:root	644
/etc/cni/net.d/calico-kubeconfig	root:root	644

metrics-server Upgrade

In the metrics-server directory, run the following command:

$ kubectl apply -f components.yaml serviceaccount/metrics-server unchanged clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged service/metrics-server unchanged deployment.apps/metrics-server configured apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged

Once the metrics-server deployment is updated, the pod will restart.

kube-state-metrics Upgrade

In this case, we’ll be applying the entire directory so from the configurations directory, apply the following command:

$ kubectl apply -f kube-state-metrics/ clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics configured clusterrole.rbac.authorization.k8s.io/kube-state-metrics configured deployment.apps/kube-state-metrics configured serviceaccount/kube-state-metrics configured service/kube-state-metrics configured

Once the kube-state-metrics deployment is updated, the pod will restart.

Filebeat Upgrade

Filebeat uses Elastic Stack clusters in four environments. Filebeat itself is installed on all clusters. Ensure you’re managing the correct cluster when upgrading the filebeat container as configurations are specific to each cluster.

Change to the appropriate cluster context directory and run the following command:

$ kubectl apply -f filebeat-kubernetes.yaml configmap/filebeat-config configured daemonset.apps/filebeat configured clusterrolebinding.rbac.authorization.k8s.io/filebeat unchanged clusterrole.rbac.authorization.k8s.io/filebeat configured serviceaccount/filebeat unchanged

Verification

Essentially monitor each cluster. You should see the filebeat containers restarting and returning to a Running state.

$ kubectl get pods -n monitoring -o wide