I’m running an OKD4 (aka an upstream Red Hat OpenShift Container Platform v4) cluster at home. Recently my bldr0cuomokdmst1 master node crapped out. I couldn’t even log in to determine the problem. For my current Kubernetes clusters, I’m casting the logs to my ELK clusters but hadn’t done that for the OKD4 cluster yet. So no idea why it failed. Now I need to delete the old master, clear out the configuration, and add it back in again.
Preparation
First off, you should have already replaced the certificate in your installation process. See my OKD4 Installation post for details on how to do that. As a reminder, the certificate is only good for 24 hours for a new cluster build. After that you need to retrieve the cluster certificate and add it to the ignition file.
High Availability Proxy (haproxy)
I removed the failed master node from the cluster in part because it was causing timeout problems with managing the cluster while I was researching the solution. Sometimes the console would work, other times I suspect it was trying to get to the failed master and timing out. However deleting the master and adding it back in is a pretty quick process so removing it from the haproxy configuration might not be necessary.
Log in to your cluster haproxy server (bldr0cuomokdhap1) and in /etc/haproxy, edit the haproxy.cfg file to comment out the bldr0cuomokdmst1 server entry from the configuration. Don’t delete it as we’ll be uncommenting it when the master has been recovered.
Remove Master
Drain the node from the cluster. While the pods won’t be removed, the node will be cleared from the system so it’s not accepting incoming connections any more.
$ oc drain bldr0cuomokdmst1 \
--delete-emptydir-data \
--ignore-daemonsets \
--force
This will take some time as requests to the failed master will need to time out. Now verify bldr0cuomokdmst1 has been removed as an endpoint in the cluster. The IP is 192.168.101.100 which should be missing from the output below.
$ oc describe svc kubernetes -n default
Name: kubernetes
Namespace: default
Labels: component=apiserver
provider=kubernetes
Annotations: <none>
Selector: <none>
Type: ClusterIP
IP Families: <none>
IP: 172.30.0.1
IPs: 172.30.0.1
Port: https 443/TCP
TargetPort: 6443/TCP
Endpoints: 192.168.101.102:6443,192.168.101.103:6443
Session Affinity: None
Events: <none>
Finally delete bldr0cuomokdmst1 from the cluster.
$ oc delete node bldr0cuomokdmst1
node "bldr0cuomokdmst1" deleted
Clear etcd
Since etcd is on the masters and not a separate cluster, we’ll need to remove the etcd configuration as well. You’ll log in to a working etcd pod, remove bldr0cuomokdmst1, then remove any secrets that belong to the master node from the cluster.
Verify Status
$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'
2 of 3 members are available, bldr0cuomokdmst1 is unhealthy
Working Pods
$ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd etcd-bldr0cuomokdmst2 3/3 Running 0 33d etcd-bldr0cuomokdmst3 3/3 Running 0 33d
Clear Configuration
In this step, you’ll log in to one of the above working pods (2 or 3) and remove bldr0cuomokdmst1 from the etcd configuration.
$ oc rsh -n openshift-etcd etcd-bldr0cuomokdmst3
Defaulting container name to etcdctl.
Use 'oc describe pod/etcd-bldr0cuomokdmst3 -n openshift-etcd' to see all of the containers in this pod.
Get the member list
# etcdctl member list -w table
+------------------+---------+------------------+------------------------------+------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
| e43c9f92fda4af5 | started | bldr0cuomokdmst3 | https://192.168.101.103:2380 | https://192.168.101.103:2379 | false |
| ac4ca03e8d200e17 | started | bldr0cuomokdmst2 | https://192.168.101.102:2380 | https://192.168.101.102:2379 | false |
| c7804a193b578f80 | started | bldr0cuomokdmst1 | https://192.168.101.101:2380 | https://192.168.101.101:2379 | false |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
You can see that bldr0cuomokdmst1 is in the configuration. Now remove it.
# etcdctl member remove c7804a193b578f80
Member c7804a193b578f80 removed from cluster 617aed10ec5206e3
Finished. You can exit the etcd pod now.
Remove Secrets
There are three secrets for each master node. You’ll need to get the list and then remove them from the cluster.
$ oc get secrets -n openshift-etcd | grep bldr0cuomokdmst1
etcd-peer-bldr0cuomokdmst1 kubernetes.io/tls 2 205d
etcd-serving-bldr0cuomokdmst1 kubernetes.io/tls 2 205d
etcd-serving-metrics-bldr0cuomokdmst1 kubernetes.io/tls 2 205d
$ oc delete secret -n openshift-etcd etcd-peer-bldr0cuomokdmst1 secret
"etcd-peer-bldr0cuomokdmst1" deleted
$ oc delete secret -n openshift-etcd etcd-serving-bldr0cuomokdmst1
secret "etcd-serving-bldr0cuomokdmst1" deleted
$ oc delete secret -n openshift-etcd etcd-serving-metrics-bldr0cuomokdmst1
secret "etcd-serving-metrics-bldr0cuomokdmst1" deleted
And the failed bldr0cuomokdmst1 server has been completely removed from the cluster.
Rebuild Master
This process follows the initial build process except with a single node. You’ll boot the server to an ISO image, update the boot line, approve the Certificate Signing Request (csr), and monitor the node.
haproxy Node
If you cleared the master server from haproxy, you’ll need to uncomment that line in the haproxy.cfg file plus you’ll need the boot node so that’ll also need to be uncommented. Log in to bldr0cuomokdhap1 and in /etc/haproxy edit the haproxy.cfg file. After updated, restart haproxy.
Start Guest Image
Update the boot settings for the image to boot to the BIOS after started. Once started, attach the Fedora CoreOS ISO, make sure the system boots to CD-ROM, and save and start the image.
In the Live Image, tab to the boot line and enter in the following parameters at the end.
coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.101.100:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.101.100:8080/okd4/master.ign
Monitor the console and you’ll see the system start up, then start downloading the image, and it’ll boot. Remove the ISO from the image after it starts and reboot the system. Initially it’ll retrieve the newest image so you may see it reboot again.
Approve CSRs
Now you need to review and approve any outstanding CSRs.
$ oc get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-k258q 11m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
There should really only be one but if more than one, feel free to investigate. Once done, approve the outstanding CSR and pods will start on the bldr0cuomokdmst1 node.
$ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/csr-k258q approved
At this point, the replacement master node is part of the cluster and is creating pods. As there are quite a few (43 at my last count) on a single master node and I’m on high speed WiFi, it will take several minutes.
Verify etcd Configuration
Once all the pods have started and the new cluster member is Ready, verify etcd is also working. First off, check the health of the cluster.
$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'
3 members are available
That looks good. Log in to the same etcd pod you did at the start and check the table output.
$ oc rsh -n openshift-etcd etcd-bldr0cuomokdmst3
Defaulting container name to etcdctl.
Use 'oc describe pod/etcd-bldr0cuomokdmst3 -n openshift-etcd' to see all of the containers in this pod
# etcdctl member list -w table
+------------------+---------+------------------+------------------------------+------------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
| e43c9f92fda4af5 | started | bldr0cuomokdmst3 | https://192.168.101.103:2380 | https://192.168.101.103:2379 | false |
| 630bbe550c81b877 | started | bldr0cuomokdmst1 | https://192.168.101.101:2380 | https://192.168.101.101:2379 | false |
| ac4ca03e8d200e17 | started | bldr0cuomokdmst2 | https://192.168.101.102:2380 | https://192.168.101.102:2379 | false |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
haproxy Update
The final task is to remove the boot server from the haproxy configuration. Log in to bldr0cuomokdhap1 and edit /etc/haproxy/haproxy.cfg. Comment out the boot server lines, save, and restart haproxy.