Well, I did chat with the landlords for the Hobbydown space, gave them a quick elevator speech. Of course they’re still looking at the nationwide spa taking it however they do feel we are the best choice but first come, first served. If it falls through, we still have a chance.
Unfortunately, no further emails from the first place but the second place called us. The 1517 space we liked is being taken by a nationwide Vet clinic. However they have another place in the same mall that’s becoming available at 1515 Main. We did a walk through last week and I sent an email over to the owners/realtors (same folks) with my requirements. The space is currently a cleaning supply service. Their lease ends end of August and he goes month to month. My lease ends January 31st.
Current floorplan. The front is to the right (or east side) with lines where the double doors and windows are. The back has a pair of lines where the back door is:
The request is to leave the north extension/wall and the vertical wall and remove the perpendicular wall to it. Plus remove the storage walls towards the north wall and against the west wall. As the floor from the vertical wall to the front of the store is concrete, replace the carpet in the back and put down new carpet across the entire floor.
I’d put glass cabinets from the east side of the north extension to the east wall, leave the space behind the wall for storage and the office for, well an office. Miniatures tables on the other side of the vertical wall and card and board gaming tables between the vertical wall and retail space.
We’ll see how this works out. But the new place does have appeal. It doubles both our retail space and the gaming space. Let’s see if they are willing to move forward with the changes.
New floor plan after walls are removed and tables in place.
Okay if you recall I bought my Friendly Local Game Store (FLGS) last year. In fact, in a couple of weeks, it’ll be exactly a year. As a reminder, I stumbled onto an ad for a Board Game Cafe up in Saskatoon Canada. Looked interesting and I spoke to my good friend, who was the owner of my FLGS along with his wife. Just to be clear, I do go to many of the surrounding game shops but Atomic Goblin Games is my F LGS. After discussion and months of review, getting financials together, getting a loan, approvals from my wife, I purchased my FLGS, which is now actually my FLGS.
I did hire the former owner as my store manager/purchasing manager. I also kept on the other employees and gave them all raises, now being paid above minimum wage (just a little though, they do get a 30% discount on games). I also brought in the former game store owner where my guy used to work, as a contractor for an occasional couple of hours of work when needed and as a mentor to help me on my way as a retail game store owner. These guys have been great, let me just say. Plus the number 1 ‘senior associate’ is in line to be assistant manager.
Anyway, we’re now bursting at the seams. Lots of gamers, lots of customers, and lots of product. Humorously last year I told the “management team” that I’d want to open a second Atomic Goblin Games Too or move to a bigger location and they were all, “well, get some experience and let’s see in a year or two where we are”.
Since I took over and since I implemented changes, over the course of the year since I purchased it, we’ve had the best 12 months in the almost 11 year history of the store, most of the last 12 months have been the best month in the 11 year history, and right now we’re better than 50% higher in sales than last year at this time.
The problem with where we are is the management companies (yes two) have not been helpful in us expanding into the mainly empty space next door. This would let us expand another 600 or so square feet, from 1,400 to 2,000 square feet mainly (I don’t have an exact number so close enough). It’s being used a storage for the end unit, a gas station/convenience store. Not only that, I’ve been trying to get the lease in my name for over a year, and a couple of places won’t let me take over the old account until the lease is in my name. I can’t even replace the carpet (3 year old carpet and it’s been 10 years) without permission from the owner due to the cost (over $10,000 amount requires approval).
Anyway, enough of that. What’s new?
Well, since I can’t get the lease and can’t expand, and must expand we’ve been checking out possibilities in the city. There are three other game stores. One is in downtown (5th Street) and two are close to each other on the corner of Main Street and 119, another major road that passes through Longmont. The downtown shop is mainly comics with games and puzzles. The two on the corner are either mainly Magic specific or Warhammer specific. We are the board game shop in town.
We had one really good choice. A bit larger than we needed but with some leeway from the owners, and they were very positive, we could grow into the space. It’s a bit over 5,000 square feet which doesn’t count the upstairs that they were throwing in for free. Unfortunately they were working with a nationwide Salon first. But we were lower initial maintenance costs, no need to drill holes in the foundation for water drainage and just needed the bathrooms upgrade and replace the carpet (which the salon would need anyway). It’s also a pretty good price in general, $12 sq/ft NNN ($7.50 sq/ft) (NNN is basically the exterior maintenance costs; parking lot, grass, trees, roof, etc).
It was the spot Hobbytown occupied so was already a gaming/hobby destination in town. Win-Win, but unfortunately again, not really available. I did send a couple of emails asking if there was any way we could slide in but in the event we couldn’t, we would still look around.
Here’s the floor plan. The lower piece they were throwing in for free as it’s inaccessible otherwise.
And here’s one of the pictures.
The nice thing is it is on Main Street, there’s a BBQ Place in the end unit, Dairy Queen and Wendy’s next door. Pretty good for the gamers and such coming into the shop.
Anyway, my wife and I went driving around town today to check out places based on several commercial listings we found on line. One is a former Big 5 Sports big box store. 10,000 square feet. We checked it out and it’s really 5,000 of retail and a second floor with an additional 5,000 of office space. Could be interesting however it’s really quite large but I will call to check.
There were several other possible places. A couple in a kind of skeevy strip mall. Not horrible in general, there are several restaurants and fast food places (like Five Guys and Chipotle’s) but I dunno, kind of an off-putting feel. Plus there were 4 open spaces, not lending itself to confidence in the customers.
A second block was better, a former K-Mart converted into several shops like a Big Lots, a Cricket, and an Arc shop. It looked good looking into the window, but when I checked on line, it’s almost 8,000 square feet.
We went into one strip mall and from the listing, it looked like the open space was taken. We continued on but my wife said there was another open space that wasn’t in the listing. We were able to locate the commercial listing on another site.
It’s a touch over 3,000 square feet which fits nicely into our expansion needs. Unit A is separated from Unit B by a wall with a double wide door like opening (it was a single use). The front part is 1,500 square feet, the back the same. With us using the front part as retail, we could use Unit B as the gaming area. More room for tables and such. Plus if we are able, we might pick up Unit C and add it as an RPG gaming area with two walls as a corridor so RPGers could have quiet to play.
The cool thing? There’s a sliding garage type door. The gap in the wall between Unit B and Unit C. Now that would be great during nicer days. Open the door for the gamers and expose them to fresh air.
The bottom of the pic faces Main Street. If you clicked the link, there’s a Taco place, Pizza Hut, and an ice cream place. It feels so much nicer, neighborhood wise with outside seating at the Taco place. It just feels like a good place to move to. At 3,000 square feet (or more if we take Unit C or even Units D and E (the office space)), it would let us grow just a little better and saner than the 5,600 or 10,000 square feet places and in 5 years, maybe then move into a much larger space.
Basically I do this when I find a recipe on line that I like and want to make sure I can find it again 🙂
Ingredients
2 Rainbow Trout fillets, boned and with skin.
1/2 teaspoon kosher salt
1/4 teaspoon black pepper
1/4 teaspoon garlic granules
1/2 cup butter (half a stick)
1/2 tablespoon fresh lemon juice
2 tablespoons minced parsley
Instructions
Rinse off the trout and pat dry with a papertowel.
Sprinkle the skin side with half of the mixture of the salt and pepper. I find doing it as a pinch makes sure it’s spread evenly.
Heat the butter in a 12 inch or so nonstick skillet. When the butter starts bubbling, swirl it around to make sure it’s spread evenly around the pan.
Add the two fillets skin side down and sprinkle the rest of the salt and pepper mixture on the fleshy side. Press on the fish to make sure the skin touches the pan.
Cook time is about 3 minutes.
When the skin side is done, sprinkle the garlic on the fillets and flip it over. Cook for about 3 minutes.
After removing the fish, put the lemon in the pan and mix it up. Then drizzle the sauce over the fish.
At times you want to quickly throw up a Kubernetes cluster for some quick test or another. While I do have several Kubernetes clusters on my Homelab once in a while you want to do a quick test or even follow a tutorial to get familiar with this or that tool.
This time I’m using a tool called kind. See the References below to find the link to the site.
Installation
First off you’ll need to have a docker server in order to install the various tools. Next you’ll have to install the kind tool.
Once kind is installed, start up your test cluster. Note that kind is installed off your home directory in the go/bin directory. Either add it to your path or add it to the command.
# go/bin/kind create cluster --name nginx-ingress --image kindest/node:v1.23.5
Creating cluster "nginx-ingress" ...
✓ Ensuring node image (kindest/node:v1.23.5) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-nginx-ingress"
You can now use your cluster with:
kubectl cluster-info --context kind-nginx-ingress
Thanks for using kind! 😊
See if the cluster is up.
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
nginx-ingress-control-plane Ready control-plane,master 3m21s v1.23.5
In order to use AWX, aka the upstream product of Ansible Automation Platform, formerly Ansible Tower, we need to have a working cluster. This article provides instructions in how to install and use AWX.
Installation
Before doing the installation, you’ll either need to install the Postgresql server or simply create the postgres user account on the storage server (NFS in my case). Otherwise the postgres container won’t start without mucking with the directory permissions.
The installation process for AWX is pretty simple. In the Gitops repo under [dev/qa/prod]/cluster/awx, you’ll apply the registry-pv.yaml first. This creates the external file system on the NFS server under /srv/nfs4/registry where the postgres container will store data.
And it worked. The PVC was allocated and the pods started.
Once the registry PV has been created, you might want to update the AWX tag in the kustomization.yaml file. The one in the Gitops repo is at 2.16.1 however as a note, when you kick it off, AWX will upgrade so as long as you’re not too far off of the next version, you can probably just apply as defined. Not it is ‘-k’ to tell kubectl this is a kustomization file and not a “normal” yaml file.
$ kubectl apply -k .
namespace/awx created
customresourcedefinition.apiextensions.k8s.io/awxbackups.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxmeshingresses.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxrestores.awx.ansible.com created
customresourcedefinition.apiextensions.k8s.io/awxs.awx.ansible.com created
serviceaccount/awx-operator-controller-manager created
role.rbac.authorization.k8s.io/awx-operator-awx-manager-role created
role.rbac.authorization.k8s.io/awx-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/awx-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/awx-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/awx-operator-awx-manager-rolebinding created
rolebinding.rbac.authorization.k8s.io/awx-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/awx-operator-proxy-rolebinding created
configmap/awx-operator-awx-manager-config created
service/awx-operator-controller-manager-metrics-service created
deployment.apps/awx-operator-controller-manager created
awx.awx.ansible.com/awx-demo created
Postgres Database and Storage
The postgres container has a default configuration that uses attached storage (PV and PVC) for the database information. This is an 8g slice. The problem is it creates a [share]/data/pgdata directory with the postgres database. This means you have to ensure you have a unique PV for each postgres container.
Of course if you’re using an external postgres server, make sure you make the appropriate updates to the configmap.
Ingress Access
In addition to the pods, we need to create a DNS entry plus an ingress route.
And now AWX should be up and running. It may do an upgrade so you may have to wait a few minutes.
$ kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-demo-postgres-13-0 1/1 Running 0 24m
awx-demo-task-857c895bf9-rt2h8 4/4 Running 0 23m
awx-demo-web-6c4df77799-6mn9p 3/3 Running 0 21m
awx-operator-controller-manager-6544864fcd-tbpbm 2/2 Running 0 2d13h
Local Repository
One of the issues with being on a high speed WiFi internet connection is we don’t want to keep pulling images from the internet. I have a Docker Directory, aka a local Docker Registry where I’ll identify images that Kubernetes or Openshift uses, pull them to my docker server, tag them locally, push them up to the local Repository, and then update the various configurations either before adding them to the cluster or updating them after they’ve been installed. The main container we want to have locally is the automation one (awx-ee:latest) as it’s created every time a job runs.
Here is the list of images found while parsing through the awx namespace. Note we also update the imagePullPolicy to Always as a security measure.
quay.io/ansible/awx:24.3.1
quay.io/ansible/awx-ee:24.3.1
quay.io/ansible/awx-ee:latest
quay.io/ansible/awx-operator:2.16.1
kubebuilder/kube-rbac-proxy:v0.15.0
docker.io/redis:7
quay.io/sclorg/postgresql-15-c9s:latest
For the local docker repository, I’ll pull the image, tag it locally, then push it to the local server. Like so:
You’ll have to get the admin password, run the following command to retrieve it. Once retrieved, log in to https://awx.dev.internal.pri (or whatever you’re using) as admin and use the password. When you log in, the password is cleared so make sure you save it somewhere.
If you forget the admin password or simply want to reset it, you would log into the web container and reset it there.
$ kubectl exec awx-demo-web-6c4df77799-6mn9p -n awx --stdin --tty -- /bin/bash
bash-5.1$ awx-manage changepassword admin
Changing password for user 'admin'
Password:
Password (again):
Password changed successfully for user 'admin'
bash-5.1$
Troubleshooting
Persistent Volumes
The one issue I had was the persistent volume claim (PVC) failed to find appropriate storage.
$ kubectl describe pvc postgres-13-awx-demo-postgres-13-0 -n awx
Name: postgres-13-awx-demo-postgres-13-0
Namespace: awx
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/component=database
app.kubernetes.io/instance=postgres-13-awx-demo
app.kubernetes.io/managed-by=awx-operator
app.kubernetes.io/name=postgres-13
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: awx-demo-postgres-13-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 3m17s (x14344 over 2d11h) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
However I have a persistent volume.
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
storage-pv 100Gi RWX Retain Available 165d
It took just a little digging but I figured out the problem.
This article lists a couple of issues that occurred while I was building this environment. The issues don’t fit into any of the specific articles mainly because it was likely due to my testing vs anything that occurred during the installation. The final article will be accurate and should just work however during the testing, these were identified and I had to track down a fix.
Terminating Pods
I had some pods that got stuck terminating for a long period of time. There were a couple of suggestions but the two that seemed to work best was to either force it or clear any finalizers for the pod. A finalizer is basically a task that needs to run and the pod is waiting for a successful completion of the task before it removes the pod.
The main solution I used for this was to force the delete. Make sure you know what node the pod is on before deleting it.
$ kubectl get pods -llamas -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
llamas llamas-f8448d86c-br4z8 1/1 Terminating 0 3d1h 10.42.232.197 tato0cuomknode3.stage.internal.pri <none> <none>
$ kubectl delete pod llamas-f8448d86c-br4z8 -n llamas --grace-period=0 --force
Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "llamas-f8448d86c-br4z8" force deleted
This can be an issue as noted, the underlying resource the pod was waiting on could also be stuck. The main thing I did was to restart kubelet on the node the pod was running on. Make sure you get the node name before forcing the deletion. Otherwise it’s safest to restart kubelet on all worker nodes, no fun if there are more than a few. If you don’t have privileged access to the node though, you’ll have to get with your sysadmin team.
Application Deletion
In working with projects in ArgoCD, I mucked up one of the projects so badly it couldn’t be deleted. This was due to some process that needed to complete but had been removed outside the normal deletion process (apparently). This time I removed the finalizer process and the application simply was deleted.
$ kubectl get application -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd llamas Unknown Unknown
$ kubectl patch application/llamas -n argocd --type json --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'
application.argoproj.io/llamas patched
$ kubectl get application -A
No resources found
The last two jobs indicated they were interested in or already doing green/blue deployments. There are multiple methods of deploying in this manner. Kubernetes has two strategies in the deployment context in order to gradually roll out a new image and roll back in the event of a problem. Argo provides a Rollout deployment process that provides additional deployment strategies via controller and CRDs.
Kubernetes Strategies
There are two update strategies that are available in Kubernetes by default. The Recreate strategy and the RollingUpdate strategy.
The Recreate strategy essentially deletes all existing pods before starting up the new pods.
The RollingUpdate strategy uses maxUnavailable and maxSurge to do an rolling update. Default values for both are 25% although you can set it to an absolute number that define the number of pods.
Argo Rollouts Strategies
There are two update strategies that are provided by the Argo Rollouts tools. The blueGreen strategy and the Canary strategy. Note the Kubernetes documentation does provide information on how you can perform a Canary update in Kubernetes. In Argo Rollouts, Canary is an actual strategy.
blueGreen Strategy
The blueGreen strategy from Argo Rollouts has two services. An Active service and a Preview service. When an upgrade is made to the Rollout context, new images are started and the Preview service points to the new ReplicaSet. You can use the Analysis task which verifies the new ReplicaSet is ready for traffic. Once tests pass, the Active service is promoted to the new ReplicaSet and the old ReplicaSet is called down to zero.
Canary Strategy
The Canary strategy lets you configure the rollout so that a small percentage of the new Replicaset is available to users. You define steps in the strategy that must complete before migrating more users to the new ReplicaSet. The steps can be simple pause statements where you wait a specific period of time before migrating more users or checks for the number of active pods. You can even have it pause until someone manually promotes the ReplicaSet.
Green/Blue Process
The problem with these strategies is they don’t address the need for testing in the same environment. Some customers don’t have access to lower environments so having a second one in all environments means new changes can be tested before they go live. So if the requirement is to have two sites up and running, one live and one test, then the process needs something different to accommodate that requirement.
For the Llamas band website, there are two namespaces, a llamas-blue and a llamas-green. If we identify llamas-blue as the live site, the ingress context would have llamas.internal.pri as the ingress URL. But for testing, we might define the ingress URL as llamas-green.internal.pri in the llamas-green namespace. In this manner, we can access https://llamas-green.internal.pri, test new features, and ensure it continues to work as expected in a production environment. When ready to activate the llamas-green namespace, we update the llamas-blueingress context for the URL to be llamas-blue.internal.pri and update the llamas-greeningress context for the URL to be llamas.internal.pri.
This gives us the capability of immediately switching back to the llamas-blue project in case something goes wrong. Otherwise all future updates now apply to the llamas-blue project.
In this article, I’ll be providing details on how to configure ArgoCD for the Llamas Band project including deploying to the other sites.
Continuous Delivery
With ArgoCD installed and the Llamas container CI pipeline completed, we’ll use this configuration to ensure any changes that are made to the Llamas website are automatically deployed when the container image is updated or any other configuration changes are made.
Project
In my homelab, there really isn’t a requirement for projects to be created however in a more professional environment, you’ll create an ArgoCD project for your application.
In this project, you’re defining what Kubernetes clusters the containers in the project have access to. I have four environments and since one project is across all four clusters, we need to configure the access under specs.destinations. When you created the links using the argocd cli, those are the same ones used in this file.
Since this is an ArgoCD configuration, it goes in the argocd namespace.
When configuring the application, since I have four sites, I’m creating each with an extension; llamas-dev for example. The project defines what sites each application can access under the specs.destinations data set.
Much of the configuration should be easy enough to understand.
name – I used the site type to extend the name
namespace – It’s an ArgoCD configuration file so argocd
project – The name of the project (see above)
repoURL – The URL where the repo resides. I’m using an ssh like access method so git@ for this
targetRevision – The branch to monitor
path – The path to the files that belong to the llamas website
recurse – I’m using directories to manage files so I want argocd to check all subdirectories for changes
destination.server – One of the spec.destinations from the project
destination.namespace – The namespace for the project
The ignoreDifferences block is used due to my using Horizontal Pod Autoscaling (HPA) to manage replicas. While HPA does update the deployment, there could be a gap where argocd terminates pods before it catches up with the deployment. With this we’re just ignoring that value to prevent conflict.
You can test this easily by setting the deployment spec.replicas to 1 (the default) then adding the HPA configuration. When checking the deployment after that, you’ll see it’s now set to 3.
$ kubectl get application -A
NAMESPACE NAME SYNC STATUS HEALTH STATUS
argocd llamas-dev Synced Healthy
Remote Clusters
Of course we also want the Dev ArgoCD instance to manage the other site installations vs installing ArgoCD to every site. Basically ArgoCD will need permission to apply the configuration files.
For the Llamas site, we’ll need a second, slightly different ArgoCD Application file. Note the difference is only the metadata.name where I added qa instead of dev, the spec.destination.server which is the api server of the cabo cluster, the spec.source.targetRevision of main instead of dev, and of course spec.source.path, the path to the Llamas files. The rest of the information should be the same.
This article provides instructions in how to build my llamas container and then how to deploy it into my kubernetes cluster. In addition, a Horizontal Pod Autoscaling configuration is used.
Container Build
The llamas website is automatically installed in /opt/docker/llamas/llamas using the GitLab CI/CD pipeline whenever I make a change to the site. I have the docker configuration files for building the image already created.
The 000-default.conf file. This file configures the web server.
<VirtualHost *:80>
ServerAdmin cschelin@localhost
DocumentRoot /var/www/html
<Directory /var/www>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
</VirtualHost>
And the Dockerfile.development file. This copies the configuration to the webserver and starts it.
FROM php:7-apache
COPY 000-default.conf /etc/apachet/sites-available/000-default.conf
COPY ./llamas/ /var/www/html.
RUN a2enmod rewrite
CMD ["apache2-foreground"]
When done, all you need to do is run the docker-compose command and your image is built.
podman-compose build
Access the running image via the docker server, port 8000 as defined in the docker-compose.yaml file to confirm the image was built as desired.
Manage Image
You’ll need to tag the image and then push it up to the local repository.
podman tag llamas_webapp:latest bldr0cuomrepo1.dev.internal.pri:5000/llamas:v1.2
podman push bldr0cuomrepo1.dev.internal.pri:5000/llamas:v1.2
Now it’s ready to be added to the Kubernetes cluster.
GitLab Pipeline
Basically whatever server you’re using as a gitlab runner will need to have podman and podman-compose installed. Once that’s done, you can then automatically build images. I’m also using tagging to make sure I only remake the image when I’m ready vs every time I make an update. Since it’s also a website, I can check the status without building an image.
You’ll use the git tag command to tag the version then use git push –tags to have the update tagged. For example, I just updated my .gitlab-ci.yml file which is the pipeline file, to fix the deployment. It has nothing to do with the site, so I won’t tag it and therefor, the image won’t be rebuilt.
Here’s the snippet of pipeline used to create the image. Remember, I’m using a local repository so I’m using tag and push to deploy it locally. And note that I’m also still using a docker server to build images manually hence the extra lines.
Apply the deployment.yaml file to deploy the llamas images. I set replicas to 1 which is the default but since HPA is being applied, it doesn’t really matter as HPA replaces the value. Under specs.template.specs I added the extra configurations from the PriorityClass article and the ResourceQuota article.
HPA lets you configure your application to be responsive to increases and decreases in how busy the site is. You define parameters that indicate a pod is getting busy and Kubernetes reacts to it and creates new pods. Once things get less busy, Kubernetes removes pods until it reaches the minimum you’ve defined. There is a 15 second cycle which can be adjusted in the kube-controller-manager if you need it to respond quicker (or less often).
In this configuration, we’ve set the CPU checks to be 50%. Once applied, check the status.
$ kubectl get hpa -n llamas
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
llamas Deployment/llamas <unknown>/50% 3 10 3 9h
Since it’s a pretty idle site, the current usage is identified as unknown. Once it starts receiving traffic, it’ll spin up more pods to address the increased requirements.
Success!
And the site is up and running. Accessing https://llamas.dev.internal.pri gives us access to the one page website.
This article provides details on my use of the GitLab Runners in order to deploy websites and then automatically build, tag, and push images to my local docker repository.
Runner Installation
I’ve been using Jenkins for most of my work but as someone who continually learns, I’m also looking at other tools to see how they work. In this case because where I was working, we were going to install a local GitLab server as such I want to dig into GitLab Runners.
We’ll need to create servers for the Runners in each of the environments. In addition, for security reasons, I’ll want separate Runners that have access to my remote server in Florida.
Configuration wise, I’ll have one Runner in Dev, QA, and Stage and two Runners in Production, one for local environments and one for access to the remote server, and two Runners in my Home environment, same with one for local installations and one for the remote server. What my Runners will do is mainly deploy websites (only my Llamas website is set up right now) to my Tool servers and remote server.
Each Runner server will have 2 CPUs, 4 Gigs of RAM, and 140 Gigs of disk.
Installation itself is simple enough. You retrieve the appropriate binary from the gitlab-runner site and install it.
rpm -ivh gitlab-runner_amd64.rpm
You will need to then register the runner with gitlab. You’ll need to get the registration token from the gitlab. Click on the ‘pancake icon’, Admin, CI/CD, Runners, and click the Register an instance runner drop down to get a Registration token. With that, you can register the runner.
I do have multiple gitlab-runner servers. This one is the development one that also processes containers. Other gitlab-runner servers test code or push code to various target servers.
GitLab CI/CD File
Now within your application, you can set up your pipeline to process your project on this new gitlab-runner. You do this in the .gitlab-ci.yml file. For this example, I’m again using my Llamas band website in part because it builds containers plus pushes out to two web sites so there’s some processing that needs to be done for each step. Let’s check out this process.
Test Stage
In the Test Stage, the gitlab-runner server has various testing tools installed. In this specific case, I’m testing my php scripts to make sure they all at least pass a lint test. There are other tests I have installed or can install to test other features of my projects. Note that I will use the CI_PROJECT_DIR for every command to make sure I’m working in the right directory.
test-job: tags: - test stage: test script: - | for i in $(find "${CI_PROJECT_DIR}" -type f -name *.php -print) do php -l ${i} done
Docker Stage
In this section, I’m building the container, retagging it to be loaded to my local registry, and then pushing it to the registry. But only if the site’s been tagged. I only tag when I’m actually releasing a site version. So if no tag for this push, the build is skipped. I do push it out to the separate docker server though. I do keep all binary information on the two dev servers, bldr0cuomdev1 and ndld1cuomdev1 in the /opt/static directory structure. And unlike the other stages, there is no need to clear out the .git files and directories as they aren’t part of the llamas directory so won’t be in the container.
The next stage cleans up the git and docker information and moves the website from the llamas directory down to the documentroot. Then the site is pushed out to the local web server for review.
The last stage pushes the website out to my remote server. As it’s effectively the same as the local stage, there’s no need to duplicate the listing.
Pipeline
This is actually at the top of the .gitlab-ci.yml file and lists the steps involved in the pipeline build. If any of these stages fails, the process stops until it’s resolved. You can monitor the status in gitlab by going to the project and clicking on CI/CD. The most recent job and stages will be listed. Click on the stage to see the output of the task.
stages: - test - deploy-docker - deploy-local - deploy-remote
Podman Issue
Well, when running the pipeline with a change and a tag, gitlab-runner is unable to build the image. Basically when run from gitlab, gitlab-runner isn’t actually logged in so there’s an error:
See when someone logs in, a socket is created in /run/user with the user id. But the gitlab-runner account isn’t actually logging in. So the uid 984 isn’t being created. I manually created it and was able to successfully use podman-compose but waiting a short time and the uid is removed by linux and rebooting caused it to disappear as well.
I did eventually find an article (linked below) where the person having the problem finally got an answer. Heck, I didn’t even know there was a loginctl command.
loginctl enable-linger gitlab-runner
From the man page:
Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller.
And it worked! Now to add that to the ansible script and give it a try.
