Motorcycle Touring

Now that I can check out files, edit, and check them back in. The last step is syncing the files with the target server or servers. I’m trying to eliminate the extra static files/put them into the repo vs having them be a second area to manage. Part of the problem is other teams. We want to be able to have them manage files without having to log in to the git server and manually touch the old static files.

It works pretty much the same as the previous configuration.

Copy the unixsvc public key to the target server(s).

Set up a script to do a pull and check for the error code.

If changes, use rsync to sync the data across.

Simple enough script. Set up a cron job to run every minute and the target server(s) will always be updated.

Need to test the heck out of this to make sure it works as expected. Add the other projects, less the inventory and status ones (they’re websites). And finish documenting it so I can enable it at work.

Next up, gitlab and jenkins. Let’s try this through a web interface using “normal” DevOps.

At least for someone like me where I’m the only person working on projects, the setup and usage of RCS and git are pretty straightforward. Once we get into team usage, it gets to be a bit more complicated. Right now the team can check out and check in a script but due to permissions, they aren’t able to sync the repositories. Fortunately the scripts do that every minute (checking for the flag file) but it’s a bit cumbersome.

Setup ssh git

There are a few bits that need to be done in order to get git set up.

1. Create the git user on the git server. Make sure you have sufficient space for all the code. I created a 30 gig slice in /opt/git and used it for git’s home directory.

useradd -c 'Git Service Account' -d /opt/git -m git
passwd git

2. You’ll need to add your public keys into git’s .ssh directory as ‘authorized_keys’. Do this for every server you will be pulling files from.

3. Create the Master repository on the git server. You won’t need to put any code into the directory but you do need to run ‘git init –base’ to initialize it.

mkdir /opt/projects
for i in suite inventory status httpd kubernetes changelog admin newuser
do
  mkdir /opt/projects/$i
  cd /opt/projects/$i
  git init --base
done

The “–base” option indicates this is a master repository, not a user’s working directory. The working directory will be on your home system.

4. On your home system, create the local or working repository.

mkdir projects
cd projects

5. If you’re creating the first repository as I would for the ‘suite’ scripts, make the ‘suite’ directory and initialize it. You’ll want to set a couple of variables as well.

mkdir suite
cd suite
git init
git config --global user.name 'Carl Schelin'
git config --global user.email cschelin@west.com

6. Since I’m converting existing RCS files, I want to bring all the previous changes into git. I’m using rcs-fast-export, a Ruby script that imports all the RCS changes into git. You’ll want to run the script in the directory.

rcs-fast-export.rb . | git fast-import && git reset

Note – this script isn’t working for the inventory application. I suspect it’s because I have three places where the same file name is used but for different purposes. Do some testing before you commit the updates.

7. Once done, push the code up to the git server. This will depend on what you have set up to manage repositories.

git push git@lnmt1cuomgit1.internal.pri:projects/suite

And you’re done. The project is ready for the team to retrieve and manage.

Team Setup

1. Very similar to above, each member of the team will need to copy their ssh keys over to the git server. Concatenate it with git’s authorized_keys file.

2. Create a projects directory. Don’t forget to set your git environment variables.

mkdir projects
cd projects
git config --global user.name 'Carl Schelin'
git config --global user.email cschelin@west.com

And you’re ready to edit code.

Managing Code

1. Before you can do anything, you’ll need to retrieve the git project. For the first time, you’ll have to use the clone options.

git clone git@lnmt1cuomgit1.internal.pri/projects/suite

This will retrieve all the files associated with the project you want to manage.

2. For subsequent updates, you’ll want to pull files from the server.

git pull git@lnmt1cuomgit1.internal.pri/projects/suite

3. You’ll now have a ‘suite’ directory. Within that are a ‘bin’ and ‘etc’ directory. Files in these directories are managed by git. As you know from RCS, you need to check out and check in changes. Use the ‘checkout’ keyword to begin editing the files. Change to the ‘bin’ directory and check out the ‘chkserver’ script.

cd suite/bin
git checkout chkserver

4. You can now edit the file. Once done, you’ll need to check it back in. It’s a two step process. You need to ‘add’ it back in and then ‘commit’ the change.

git add chkserver
git commit chkserver

Your editor of choice will display the current ‘git status’ as comments. Anything that’s not a comment will be added to the gitlog.

5. Once done, you’ll need to upload changes to the master.

git push git@lnmt1cuomgit1.internal.pri:projects/suite master

git Commands

List of commands that you’ll find useful. I’ll add more as I explore.

• git status – Show the status of the project
• git log – Show the commit log for the project or if you pass a file name, shows the commit log for the file.

As a Unix Systems Administrator, I’m a long time user of Revision Control System (RCS) to manage configuration files. My first time was in managing DNS Zone Files at NASA Headquarters. Over the past few years, I’ve been using RCS to manage my personal projects and work shell scripts. While I’m not the only one writing scripts and code, I believe I write the bulk of them. Anyway, I want to bring the team on board with managing scripts. Adding theirs into revision control and making it easy for the team to manage their and my scripts. Then we, everyone on the team and any new team members, can do nothing but benefit from managing each others scripts.

In order for me personally set up, I need to come up with a git/rcs Rosetta Stone. Not just commands but concepts. Taking the hacks I currently do to make RCS work in a team environment and bringing the team on board with documentation they can understand and use. This is done because I’m the one mainly using revision control for the scripts and I want to keep the history of the projects. Honestly though, it’s not super important to maintain the current history. Moving straight over to git with the existing files would also work fine and if problems occur, that’s what may happen. We’ll see as we progress.

There are plenty of git books and documentation but a google search doesn’t really identify a tutorial for moving from managing files in RCS to managing files in git. And while RCS is pretty simple in general, I do have a few scripts to help in managing my coding environment. Since it’s RCS, there are additional hacks to make it work the way I want it to work which make it more difficult for others on the team to manage it.

Background, environment setup first. Then some quick references in the rcs commands I’m using, followed by the scripts and data files I wrap around the rcs commands and a list of other scripts I use to manage the environment.

First off, you can poke at the various RCS books on line without too much trouble to see what RCS is and how it works. In general you have a directory of files. You can store versions in the current directory or create an RCS subdirectory and the RCS commands will store versions there. I prefer this method as it makes it cleaner when you’re viewing directories.

The environment:

code Directory

changelog
httpd
inventory
kubernetes
status
suite

makechangelog
makehttpd
makeinventory
makekubernetes
makestatus
makesuite

manifest.changelog
manifest.httpd
manifest.inventory
manifest.kubernetes
manifest.status
manifest.suite

* code – A set of directories that contain checked out scripts and RCS subdirectories.
** [project] – The source code for that project
** make[project] – A script that creates the provisioning directory structure using the manifest and copying all the files from the static directory.
** manifest.[project] – A file that contains all the scripts that belong to the project. I use a couple of symbols to create directories and list what’s being done.

archive Directory

Archived data. Either code bits that aren’t useful any more (in a [project] directory) or older data files. I like to maintain the data imports for historical purposes.

static Directory

The most current files that aren’t code. Spreadsheet .csv files for importing, pictures, data files, etc.

stage Directory

changelog
httpd
inventory
kubernetes
status
suite
tcpdf

exclude.changelog
exclude.httpd
exclude.inventory
exclude.kubernetes
exclude.status
exclude.suite
exclude.tcpdf

syncchangelog
synchttpd
syncinventory
synckubernetes
syncstatus
syncsuite

* stage – The provisioning staging area. The make[project] script copies the code and all the static data for the project into this directory under a [project] subdirectory.
** [project] – The staged files for the project.
** exclude.[project] – Files and directories that aren’t to be synchronized.
** sync[project] – The script that uses rsync to synchronize the directory structure (code and static data) out to the various servers as required.

scripts Directory

A project’s working directory for

[project] – An individual’s working directory for any scripts.
* /var/www/html – My php working directory for three projects.

The sync[project] scripts are run every minute out of cron. They’re looking for a sync.[project] file which was created by the make[project] script.

The make[project] scripts are run every night at 1am. This ensures the scripts are up to date even if a sync wasn’t performed earlier.

The rcs commands I use:

* co – With the -l(ock) option, check out the current revision, place it in the current directory, and lock the revision. This prevents others from updating the script.
* ci – With the -u(nlock) option, check in, don’t lock the script, and leave the original in place for further editing.
* rlog – Show the history of the script.

There are a lot more options and commands but as it’s just me, I haven’t needed to explore too far. These three commands do everything I need.

The script wrappers I use:

I have several scripts I use to manage the environment. There are things I want to do to make sure work is complete and that all files are included when provisioning.

* check – This is a wrapper around the various check scripts. It greps out the comments from each check script and then lists all the scripts.
* checkdiff – Compares the passed script name with the master script to show you the differences between the two.
* checkin – Runs a ‘checkdiff’ command to show the differences between the two scripts, then runs the RCS ci -u command. As ci doesn’t show differences, I wanted to be able to see what had changed so I could properly document the change.
* checkinstall – Runs through the working directory and returns a file name if a script exists in the working directory but not in the master code directory.
* checkmanifest – Parses the manifest file and reports any files that are checked out and being worked on in the working directory.
* checkout – Sees if there’s a difference between the master script and the current script. If so, it simply exits. Otherwise it checks out the script and copies it into your working directory.
* checkrw – Basically checks for any script in the code directory that has ‘rw’ permissions indicating it’s been checked out. I use it to make sure I haven’t missed any scripts when checking in several.

In the code directory, I have a few scripts that help manage the code.

* findcount – This script essentially creates a list of all files in the project and writes it to a countall file. It also counts lines of code, comments, etc for statistical purposes.
** countall – The list of all files in the project.
** countall.backup – The make[project] script runs a diff against the countall and countall.backup files. If there’s a difference, the script exits without creating the staging directory. To correct, just copy the countall file over the countall.backup file.
** fixsettings – This script makes sure the settings.php configuration file exists in every directory.
** searchall – This script does a search of every file in the working directory for the passed keyword. Helpful when searching for all instances of a keyword.

Where do the files go?

* Inventory – The inventory project goes to three servers in /usr/local/httpd/htsecure.
* php Scripts – This goes to four servers including the Ansible server to build host files. In /usr/local/httpd/bin.
* Shell Scripts – These scripts go to just the Jumpstart server and then is sync’d across all 1,200 servers.
* Kubernetes – These scripts go to each of the Kubernetes clusters in /var/tmp/kubernetes.

As you can see, there are several scripts and the environment is really configured for one person.

Goal is to document how to create your own working server using VirtualBox. One problem with a tool like Vagrant is the work environment doesn’t permit access to outside sites without going through a proxy. So setting up an environment that can be used at work will be beneficial.

1. Create your working environment using VirtualBox. You need a working directory plus a web site for testing the two web projects.
2. Create a Source Code Control git server. I’m using ssh to retrieve projects.
3. Pull the project code to your working server.
4. Check out project code.
5. Check in project code.
6. Push the project code back to the git server.
7. Provision project code to the various servers as listed above.

But the documentation needs to reference the existing environment in order for the new commands and processes to be understood.