Overview
While not really a ‘Disaster’, I am forced to rebuild my servers. Fortunately I have backups plus on the blog, a set of instructions for my Kubernetes and Openshift setups. This article will provide information on getting a DR site up and what the steps need to be in order to quickly get going.
Environment
I have 150 or so servers that do a bunch of things. About 20 of those are experimental stuff or little things I spun up a VM for in order to test or try something out. Do they need to come up? Probably. Do they need to come up first, or even soon? Nah.
Begin
Once I got my Proxmox servers up, templates created, and servers created, it’s time to determine which servers need to be in wave one.
Wave One
There are two main types of servers that need to be configured.
- Tool Servers – These are the Jump Servers used to access all the other servers. These also run the Ansible playbooks that configure every system. Some servers will require manual bits but automation should be attempted for all the others.
- Name Servers – Since all the servers actually resolve through the Name Servers, these also need to be brought up so all the other servers can successfully start.
Wave Two
I’m looking at the CI/CD pipeline to be set up next. This consists of a gitlab server, gitlab runners, jenkins servers, git servers, and development servers. At least for now, we’re pulling binary information from the development servers.
We do have a Nexus server so we’ll be investigating that to hold all the binaries in the future. In addition, on Kubernetes we have an AWX set of containers which is the upstream code for the Ansible Automation Platform. We’ll get this going when we work on Kubernetes.
- git servers – This holds the Ansible playbooks and configurations.
- gitlab server – This is the repository of all the playbooks. While I do copy to my github account, that server doesn’t always have the most complete list of projects.
- gitlab runners – These are really only used for testing and building containers for Kubernetes and Openview, however they are part of the CI/CD pipeline so should be built.
- jenkins servers – These systems manage the building of sites and deployment of Ansible playbooks to the various Tool Servers.
- development servers – These two systems have all the binary files in the /opt directory structure. Jenkins retrieves them when building out the final product.
Wave Three
At this point I’m looking at getting the Kubernetes servers up plus the supporting servers.
- NFS Servers – The Persistent Storage for the applications.
- HAProxy Servers – Ingress Routers
- Control Servers – Kubernetes API Binaries
- Worker Servers – Location for the containers
Once this is up, we can install the main containers.
- AWX – Ansible Automation
- ArgoCD – Gitops Tool
- Ingress – The internal Ingress routers.
Wave Four
At this point, we’re looking at the Personal Servers. Tool Server, git and development servers, and other such as media and backup servers.