This is a comment I heard a few years back from a manager on the Engineering side of the house. Of course it’s stated in a bad way, but the words behind it is that to advance in technology, your next step should be in Engineering (or Development) and you should be working towards that.
As a Systems Admin for many many years, I’ve found that I’m good at the job. Managing systems and providing solutions in Operations to improve the environment. It also plays to my own strengths and weaknesses. Troubleshooting problems, fixing them, and moving on vs spending all my time on a single product as a programmer or engineer. I’ve had discussions with prior managers about advancement both in Engineering and in Management. I’ve even attended Management Training. After discussion, it was felt that I should work with my strengths.
A few years back, I was moved into an Operations Engineer role. The company started implementing a Plan, Build, Run model and I was moved into the Build team. No real reason was given for the choices. The concept was to move a portion of the team away from day to day maintenance duties in order to focus on Business related Product deployments. It’s taking too long for the entire team to get Products into the field, what with being on call and dealing with maintenance issues like failing hardware so let’s pull three people away from the team to be dedicated on Business related Projects.
Sadly there was no realization that the delays are in part due to multiple groups in Operations having their own priorities. Delays were more due to the inability of the groups to fully align on a project. With three business groups each with their own priorities, the time to deploy really didn’t change much and there were fewer people to do the work.
As we moved forward, the company redefined the Build role. Titles change and we’re Enterprise Systems Engineers. One of the new goals was to move away from server builds and into server automation. Build one or two servers for the project in order to create automation and then turn the rest of the project as a Turn Key solution, over to the Run team to complete.
In addition, a new project came along and was designed to follow the Continuous Integration/Continuous Delivery (CI/CD) process. The concept is to deploy software using an incremental process vs the current 6 month to 18 month process.
The current process is to gather together all the change requests that identify problems in the product plus the product enhancement requests and spend time creating the upgraded product. It can take months to get this assembled, get code written and tested, fix problems that crop up, get it through QA testing, and then into Operations hands. And as noted before, deployment itself can take several months due to other projects on my plate and tasks on other groups plates.
The CI/CD process means that there’s a pipeline between Development into Operations. A flow, picture a stream. A problem is discovered, it’s passed back to development, they fix it, put it into the flow, automated testing is performed to make sure it doesn’t break anything, it’s put into the production support environment, then production. Rather than taking months to assemble and fix a large release, you have a single minor release and you can identify, fix, and deploy it into production in a matter of minutes. Even better, the Developer or Engineer would be on call for their product. If something happened, they’d be paged along with the teams, could potentially fix it right then, put it into the flow and let the fix get deployed. Automated testing is key here though and part of the fix is to fix or add new tests to make sure it doesn’t happen again or even with other products.
This process is pretty similar to how a single programmer working on a personal project for example might manage their program(s). I have a project I’ve been working on for years and has reached about 140,000 lines of code (actually about 98,000 lines of actual program with the remaining 42,000 lines being comments or blank lines). I’ve created a personal pipeline, without automated testing though. And an identified problem can be fixed and deployed in a few minutes. New requests can generally take about 30 minutes to complete and push out to the production locations.
This CI/CD oriented project was interesting. Plus it included creating a Kubernetes environment. I’d had someone comment about DevOps several years earlier so I’d been poking about at articles and even reading The Phoenix Project, a book about CI/CD and DevOps. A second project followed using CI/CD and Kubernetes. A part of the CI/CD process was utilizing Ansible and Ansible Tower, a tool I’ve been interested in using for a few years.
As a Systems Administrator, one of our main responsibilities is to automate. If you’re doing a task more than once, automate it. Write a script. I started out hobby programming, then working part time and then full time, back in the early 80’s. Even though I moved into system administration, I’ve been hobby programming and writing programs to help with work ever since. I believe it’s been helpful in my career as a Server Administrator. I have deeper knowledge about how programs work and can write more effective scripts to manage the environment.
In the new Build environment, it’s been even more helpful. It lets me focus even more on learning automation techniques with cloud computing (I’ve created Amazon Web Services and Google Cloud Services accounts) and working towards automating server builds. Years back I started creating “Gold Image” virtual machine templates. I also started in on scripting server installations. The rest of the team has jumped in and we have an even better installation script framework. Lately I’ve been working on a server validation script. For the almost 1,000 systems we manage, the script is reporting on 85,000 successful server checks with errors called out that can be corrected incrementally by the team (a local login report plus a central reporting structure). As part of this, I’ve started creating Ansible Playbooks and added scripts to our Script framework to make changes to systems. Eventually we’ll be able to use this script to create an automation process to make sure servers are built correctly the first time.
It’s been fun. It’s a challenge and sort of a new way of doing things. I say “sort of” because it still falls into the “automate everything” aspect of being a Systems Admin. With Virtual Machines and Docker Containers, we should be able to further automate server builds. With scripts defining how the environment should look, we should be able to create an automatic build process. Plus we should be able to create a configuration management environment as well. A central location defining a server configuration that ensures the server configuration is stable. You make a change on the server and the configuration script changes it back to match the defined configuration. Changes have to be made centrally.
DevOps though. It seems like this is really “Development learns how to do Operations”. “If you were smart, you’d be in Engineering.” Many of the articles I read, many of the posts on the various forums, many of the discussions I’ve followed on Youtube and in person really seem to be focused on Development becoming Operations. There doesn’t seem to be a lot of work on how to make Operations part of the process. It seems to be assumed that a Systems Admin will move into Engineering or Development or will move on to an environment that still uses the old methods of managing servers. Alternately we’ll be more like COBOL programmers. A niche group that gets called in for Y2K like problems.
I like writing code and I’m certainly pursuing this knowledge. I have a couple of ESX servers at home with almost 100 VMs to test Kubernetes/Docker, work on how to automate server deployments, work on different pipeline like products like Jenkins and Gitlab. I use it for my own sites in an effort to get knowledgeable about such a role and am starting to use them at work. Not DevOps as that’s not a position or role. That’s a philosophy of working together. A “DevOps Engineer” is more along the lines of an Engineer working on pipeline like products. Working to get Product automated testing in place but also working on getting Operations related automated testing in place.
If you’re not taking time to learn this on your own and the company isn’t working on getting you knowledgeable, then you’re going to fall by the wayside.