Gitlab Troubleshooting

Up here in the mountains, we get an occasional power outage. I have the servers on UPSs but generally only have 5 minutes or so to snag my laptop, log in, and start shutting down VMs before the power fails. And sometimes I’m not even around so servers basically crash.

In this case, gitlab failed to start. When checking the gitlab-ctl status output, redis is identified as down.

-bash-4.2# gitlab-ctl status
run: alertmanager: (pid 4395) 215617s; run: log: (pid 2085) 231156s
run: gitaly: (pid 4416) 215615s; run: log: (pid 2084) 231156s
run: gitlab-exporter: (pid 4446) 215614s; run: log: (pid 2088) 231156s
run: gitlab-kas: (pid 4600) 215604s; run: log: (pid 2093) 231153s
run: gitlab-workhorse: (pid 4623) 215601s; run: log: (pid 2079) 231156s
run: grafana: (pid 4658) 215598s; run: log: (pid 2086) 231156s
run: logrotate: (pid 6103) 2852s; run: log: (pid 2090) 231155s
run: nginx: (pid 4731) 215580s; run: log: (pid 2080) 231156s
run: node-exporter: (pid 4750) 215578s; run: log: (pid 2077) 231156s
run: postgres-exporter: (pid 4757) 215578s; run: log: (pid 2078) 231156s
run: postgresql: (pid 23262) 26s; run: log: (pid 2083) 231156s
run: prometheus: (pid 4782) 215576s; run: log: (pid 2075) 231156s
run: puma: (pid 23226) 28s; run: log: (pid 2087) 231156s
down: redis: 0s, normally up, want up; run: log: (pid 2082) 231156s
run: redis-exporter: (pid 4816) 215574s; run: log: (pid 2076) 231156s
run: sidekiq: (pid 23237) 27s; run: log: (pid 2081) 231156s

Okay. So I checked the logs and ran the gitlab-redis-cli stat but again got an error.

-bash-4.2# gitlab-redis-cli stat
Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: Connection refused

The socket does exist but since redis is down, there’s nothing to connect to. After a bit more sleuthing, I tried a gitlab-ctl reconfigure but that kicked out an error as well.

[2024-03-17T13:03:34+00:00] FATAL: RuntimeError: redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (redis::enable line 68) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout:
stderr: Error: Connection reset by peer

At this point, redis is the problem. I did a gitlab-ctl tail to see the logs and redis keeps trying to start but kicks out a rdb error.

2024-03-17_13:09:09.00014 25036:C 17 Mar 2024 13:09:08.999 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2024-03-17_13:09:09.00020 25036:C 17 Mar 2024 13:09:08.999 # Redis version=6.2.8, bits=64, commit=423c78f4, modified=1, pid=25036, just started
2024-03-17_13:09:09.00022 25036:C 17 Mar 2024 13:09:08.999 # Configuration loaded
2024-03-17_13:09:09.00208 25036:M 17 Mar 2024 13:09:09.001 * monotonic clock: POSIX clock_gettime
2024-03-17_13:09:09.00394                 _._
2024-03-17_13:09:09.00397            _.-``__ ''-._
2024-03-17_13:09:09.00398       _.-``    `.  `_.  ''-._           Redis 6.2.8 (423c78f4/1) 64 bit
2024-03-17_13:09:09.00399   .-`` .-```.  ```\/    _.,_ ''-._
2024-03-17_13:09:09.00400  (    '      ,       .-`  | `,    )     Running in standalone mode
2024-03-17_13:09:09.00401  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 0
2024-03-17_13:09:09.00402  |    `-._   `._    /     _.-'    |     PID: 25036
2024-03-17_13:09:09.00406   `-._    `-._  `-./  _.-'    _.-'
2024-03-17_13:09:09.00407  |`-._`-._    `-.__.-'    _.-'_.-'|
2024-03-17_13:09:09.00408  |    `-._`-._        _.-'_.-'    |           https://redis.io
2024-03-17_13:09:09.00409   `-._    `-._`-.__.-'_.-'    _.-'
2024-03-17_13:09:09.00410  |`-._`-._    `-.__.-'    _.-'_.-'|
2024-03-17_13:09:09.00411  |    `-._`-._        _.-'_.-'    |
2024-03-17_13:09:09.00411   `-._    `-._`-.__.-'_.-'    _.-'
2024-03-17_13:09:09.00412       `-._    `-.__.-'    _.-'
2024-03-17_13:09:09.00413           `-._        _.-'
2024-03-17_13:09:09.00417               `-.__.-'
2024-03-17_13:09:09.00417
2024-03-17_13:09:09.00418 25036:M 17 Mar 2024 13:09:09.003 # Server initialized
2024-03-17_13:09:09.00419 25036:M 17 Mar 2024 13:09:09.003 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2024-03-17_13:09:09.00443 25036:M 17 Mar 2024 13:09:09.004 * Loading RDB produced by version 6.2.8
2024-03-17_13:09:09.00445 25036:M 17 Mar 2024 13:09:09.004 * RDB age 264959 seconds
2024-03-17_13:09:09.00448 25036:M 17 Mar 2024 13:09:09.004 * RDB memory usage when created 6.59 Mb
2024-03-17_13:09:09.05154 25036:M 17 Mar 2024 13:09:09.051 # Short read or OOM loading DB. Unrecoverable error, aborting now.
2024-03-17_13:09:09.05159 25036:M 17 Mar 2024 13:09:09.051 # Internal error in RDB reading offset 0, function at rdb.c:2750 -> Unexpected EOF reading RDB file
2024-03-17_13:09:09.08898 [offset 0] Checking RDB file dump.rdb
2024-03-17_13:09:09.08905 [offset 26] AUX FIELD redis-ver = '6.2.8'
2024-03-17_13:09:09.08906 [offset 40] AUX FIELD redis-bits = '64'
2024-03-17_13:09:09.08907 [offset 52] AUX FIELD ctime = '1710415990'
2024-03-17_13:09:09.08908 [offset 67] AUX FIELD used-mem = '6911800'
2024-03-17_13:09:09.08908 [offset 83] AUX FIELD aof-preamble = '0'
2024-03-17_13:09:09.08909 [offset 85] Selecting DB ID 0
2024-03-17_13:09:09.08910 --- RDB ERROR DETECTED ---
2024-03-17_13:09:09.08910 [offset 966671] Unexpected EOF reading RDB file
2024-03-17_13:09:09.08911 [additional info] While doing: read-object-value
2024-03-17_13:09:09.08912 [additional info] Reading key 'cache:gitlab:flipper/v1/feature/ci_use_run_pipeline_schedule_worker'
2024-03-17_13:09:09.08913 [additional info] Reading type 0 (string)
2024-03-17_13:09:09.08913 [info] 4828 keys read
2024-03-17_13:09:09.08914 [info] 3570 expires
2024-03-17_13:09:09.08915 [info] 42 already expired

Unexpected EOF reading RDB file. The final solution is to delete the /var/opt/gitlab/redis/dump.rdb. I’m not a fan of just deleting the file so I backed it up and restarted redis.

-bash-4.2# mv dump.rdb ~
-bash-4.2# gitlab-ctl stop redis
ok: down: redis: 0s, normally up
-bash-4.2# gitlab-ctl start redis
ok: run: redis: (pid 26114) 1s
-bash-4.2# gitlab-ctl status
run: alertmanager: (pid 4395) 216115s; run: log: (pid 2085) 231654s
run: gitaly: (pid 4416) 216113s; run: log: (pid 2084) 231654s
run: gitlab-exporter: (pid 4446) 216112s; run: log: (pid 2088) 231654s
run: gitlab-kas: (pid 4600) 216102s; run: log: (pid 2093) 231651s
run: gitlab-workhorse: (pid 4623) 216099s; run: log: (pid 2079) 231654s
run: grafana: (pid 4658) 216096s; run: log: (pid 2086) 231654s
run: logrotate: (pid 6103) 3350s; run: log: (pid 2090) 231653s
run: nginx: (pid 4731) 216078s; run: log: (pid 2080) 231654s
run: node-exporter: (pid 4750) 216076s; run: log: (pid 2077) 231654s
run: postgres-exporter: (pid 4757) 216076s; run: log: (pid 2078) 231654s
run: postgresql: (pid 26191) 2s; run: log: (pid 2083) 231654s
run: prometheus: (pid 4782) 216074s; run: log: (pid 2075) 231654s
run: puma: (pid 26123) 27s; run: log: (pid 2087) 231654s
run: redis: (pid 26114) 28s; run: log: (pid 2082) 231654s
run: redis-exporter: (pid 4816) 216072s; run: log: (pid 2076) 231654s
run: sidekiq: (pid 26162) 10s; run: log: (pid 2081) 231654s

And that seems to have done the trick for redis. I ran gitlab-ctl stop to completely stop gitlab and then rebooted the server.

Once up though, postgresql failed to start. In checking, I received the following error

PANIC:  could not locate a valid checkpoint record

It took a bit of hunting to find a basic solution. Basically I needed to reset by running the pg_resetwal program in /opt/gitlab/ but by becoming the gitlab-psql user. But first I had to stop gitlab and then become the gitlab-psql user and run the program to reset the record. It’s not a great solution but as a result of the server resetting due to a power outage.

-bash-4.2# su - gitlab-psql
Last login: Sun Mar 17 17:52:00 UTC 2024 on pts/0
-sh-4.2$ pg_resetwal -f /var/opt/gitlab/postgresql/data
Write-ahead log reset

Then I started the server again and checked the status.

-bash-4.2# gitlab-ctl status
run: alertmanager: (pid 22057) 35s; run: log: (pid 1780) 16764s
run: gitaly: (pid 22068) 35s; run: log: (pid 1801) 16764s
run: gitlab-exporter: (pid 22085) 34s; run: log: (pid 1781) 16764s
run: gitlab-kas: (pid 22087) 34s; run: log: (pid 1814) 16763s
run: gitlab-workhorse: (pid 22098) 33s; run: log: (pid 1798) 16764s
run: grafana: (pid 22108) 33s; run: log: (pid 1779) 16764s
run: logrotate: (pid 22118) 32s; run: log: (pid 1808) 16764s
run: nginx: (pid 22125) 32s; run: log: (pid 1796) 16764s
run: node-exporter: (pid 22133) 32s; run: log: (pid 1787) 16764s
run: postgres-exporter: (pid 22139) 31s; run: log: (pid 1785) 16764s
run: postgresql: (pid 22147) 31s; run: log: (pid 1799) 16764s
run: prometheus: (pid 22150) 30s; run: log: (pid 1786) 16764s
run: puma: (pid 22166) 30s; run: log: (pid 1778) 16764s
run: redis: (pid 22172) 29s; run: log: (pid 1797) 16764s
run: redis-exporter: (pid 22178) 29s; run: log: (pid 1784) 16764s
run: sidekiq: (pid 22185) 29s; run: log: (pid 1800) 16764s

Resources:

  • https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/293
  • https://forum.gitlab.com/t/postgresql-down-after-upgrade-from-13-10-to-13-12/57018
Posted in Computers, gitlab | Tagged , , | Leave a comment

Ansible Automation Platform Projects

Overview

This article will provide instructions in how to configure Ansible Automation Platform (AAP) and how to get your Project working. Links for various fields that I don’t need in my environment is provided at the end of this article.

Organizations

Before you can do anything, you need to create an Organization as all additional information is associated with an Organization.

In the AAP console, click on Organizations, then Add.

  • Name – This is selected for every additional task done in AAP.
  • Description – A reasonable description of the Organization.
  • Instance Group – A collection of Container Instances. This lets you run jobs on specific isolated containers. Otherwise all jobs run on the AAP container.
  • Execution Environment -This replaces the Python virtual environment. This gives you a customized image to run jobs where some dependency isn’t impacted by the requirements of a different job.
  • Galaxy Credentials

User Management

While the Admin can make changes, the Admin should really only be administering the cluster and not dealing with the day-to-day work.

The next step then is to add the users. Under Access, click Users and fill in the form. Under User Type, there are three. Normal User, System Auditor, and System Administrator.

  • Normal User – General access used by all members
  • System Auditor – Read-Only access to the entire environment
  • System Administrator – Admin access to the cluster.

Group Management

In AAP, Teams manage access to the various parts of an Organization. When a team member creates something such as Credentials, only that team member can use that Credential. If the Credential is to be used by all, then the Team would need to be given permission to access that Credential. Then all members of the team have access.

The next step is to create the necessary team(s). Under Access, click Teams and create the new team.

To add members to the team, under Access, click the Users link. Select the user you want to add. Under the Teams tab, select the team(s) the member should belong to.

GitLab Personal Access Token

Next up we need to access Gitlab in order to pull a repository into AAP. To do so, we need to create a Group Access Token. In Gitlab, under the group account, External-Unix in my case, your account and preferences, click on the Access Tokens link and create the token. I don’t need AAP to write back to the repository as it’s just applying configuration information for Kubernetes. So just select the read_respository option and a reasonable name and expiration date.

In AAP , click on the Credentials link and click the Add button. Add a Name, I used Gitlab Access Token, a Description, and of course the Organization you created. Under Credential Type, select Gitlab Personal Access Token. While it’s actually a Group Access Token, there isn’t an option for that in the menu. When you select it, a Token field is displayed. You’ll add the Group Access Token here.

Machine Credential

I’m using a SSH private/public RSA key and a service account to run ansible playbooks. My service account has passwordless access to sudo to root on all servers so I don’t need to pass in a service account password.

I need to create a Machine Credential for my Service Account. In the Credentials link, select the Machine Credential Type. Several fields are now available. If you use password access, you will fill in the necessary information. If you’re using a SSH Key, you can enter your passphrase here as well. In my case, I add my SSH Private Key for the service account and it’s ready to go.

Source Control Credential

In order to pull code from GitLab using the ssh link, we need to create a Source Control Credential. Click Add to create a new Source Control Credential. Similar to the Machine Credential, fill out the fields and then add the private ssh key used when you push git updates to GitLab.

Projects

Now that all the Credentials have been created, we can access the repository and bring in the Ansible playbooks. First though, you’ll need the ssh link to your repository. In GitLab, navigate to that repository, click on Clone, and copy the Clone with SSH link.

In AAP, click on Projects and Add. Enter a Project Name and Description. Select the Organization. If you’re validating content, select the appropriate Content Signature Validation Credential. And for the Source Control Type, select Git. This brings up several additional fields specific to the Git selection. Paste the SSH Clone Link into the Source Control URL field. Under the Source Control Branch/Tag/Commit field, add in the branch, tag, or commit id you want to use for the Project. And under the Source Code Credential field, select the SSH Access to GitLabcredential you created earlier.

Branch/Tag/Commit

A quick aside here on this field. If you are deploying to multiple environments, you might consider different strategies for the Projects. For example, the Development environment might be better using a branch strategy as every change then gets applied to the development server(s) for review. For pre-Production environments such as a QA or Staging environment, you might use a git tag to lock in a release. And for Production, you’d use a commit id. This locks Production so even an accidental push to the repository won’t cause Production to update until you’re absolutely ready.

Inventories

One of the things I want to do is have git manage the inventory files, for my environment that’s GitLab. There are three options for an Inventory but for inventories managed in a repository, select just Inventory from the drop menu and fill out the Name, Description, and Organization, then Save the Inventory.

Once the inventory is created, select it. There are several tabs, one is Sources. Click the Sources tab and click on Add. Enter a Name and Description for the inventory, then select Sourced from Project from the Source drop down. Additional fields will be displayed.

Under Project, select the Playbooks project. If there, select inventory/hosts from the Inventory file drop down. If it’s not there, you can manually enter it. Check the Overwrite checkbox to clear old systems that have been retired or otherwise removed from the hosts file. Upon the next sync, they should disappear from the list of Hosts. Verify it by checking the Hosts link. If they still exist, you can click the slider next to the host to remove it from consideration until AAP clears the entry.

I will note that my inventory files are automatically created by my Inventory web application. This file is then copied into the inventory directory of the Playbooks repository. Since it’s unlikely to happen often, you don’t have to set up a job to sync regularly.

Click Save and then either click the Sync all button or if more than one inventory exists, click the circling arrows to the right of the Inventory. Depending on the number of hosts, it may take a few minutes for all the hosts to register.

Templates

Finally in order to run your playbooks, you need to create a Template. Select Templates and click on Add. There are two options on the drop down menu; a Job Template for a single task or a Workflow Template for multiple tasks such as a pipeline. For this article, select a Job Template.

Several fields are now available to create the template. Enter in a Name and Description. A Job Type drop down lets you Run the job or Check the job before running (the -C or –check on the command line). Select the Inventory to use, the Project, and within the Project, the Playbook you want to run. For Credentials, I’m selecting the Machine ssh one as it’s my service account that has authority to run the playbooks.

For this article, I’m using my Unixsuite playbook which ensures my standard scripts are on all my servers. I do want to have each environment run separately though so I’m creating multiple Templates. I typically pass the variable as ‘pattern=[env]’. Since this isn’t the command line, I’ll have to add it in the Variables box. My QA environment uses the ‘cabo0’ prefix and pattern is the variable in the playbook so the following should be entered in the Variables box:

---
pattern: "cabo0"

Once saved, click the Template, then click the Launch button to make sure it works. Once you’re sure it’s working as expected, go back to the Templates and select the newly created Template. Click on Schedules and Add. Create a Name and Description. Select a Repeat frequency, I selected weekly. This brings up a few more fields where you can further customize the schedule. I selected Saturday and Never for when it Ends. Then a Start date/time. Since I’m doing it weekly on Saturdays, I selected 11:15pm. Then Save it.

Teams Management

Now that you’ve created all these entries to have your project run regularly, no one else on the team will be able to manage the bits you’ve created. Not until the Team has access.

Under Access, click Teams and then click on the Team you want to modify. Click the Roles tab to begin to assign Roles.

You’ll be presented with multiple options for what can be added to the team. I’ve selected Credentials in this case as I want the team to be able to access GitLab. Click Next to select the Credentials to add.

As you can see, the last three are the Credentials we created in this article. Click the three checkboxes and click Next.

Finally select the rights the team has to the selected credentials. In general Use access is sufficient and assumes you want no one else to manage the selected credentials. Once you’ve selected the rights, click Save and all members of the team have access.

You’ll want to do the same thing with other permissions. Access to inventories, playbooks, templates, and so on are granted through the teams interface. By default only the creator has access to the tasks they created.

References

This section provides links to the Ansible documentation. The steps in this article provide the order in which to configure my Ansible Automation Platform (AAP) installation. The links provide more detail that might be beyond the scope of my requirements.

Posted in ansible, Computers, Git | Tagged , , | 1 Comment

Game Store Move: Todo List

I created a shared Google Doc document and shared it with the team. I am popping over and updating this list but the Google Doc is the final doc.

Work Estimates

  • Asbestos Testing. I called Rex Environmental. 5 Samples need to be taken. Same day testing, 24-48 hour results.
  • Architectural Drawings. The city requires drawings of the before and after. I called F9 Productions Inc he did suggest calling the City Inspector to see if it was needed. The City FAQ says yes. I did get a fast reply, no drawings are needed.
  • Demolition Permit. Longmont charges $50 for the permit.
  • Demolition. I called Gorilla Demolition.
  • Carpeting. I called Family Carpet One Floor & Home. These folks did the current shop’s carpet. After a review of the space, we’ll put carpeting in the retail space and Luxury Plank in the gaming space.
  • Shop Signage. I called Rabbit Hill Graphics and got a very nice sign.
  • Moving Company. I called Johnson Moving and Storage as we used them to move to the house.
  • Electrical Update. The shop needs 12 new outlets. The north wall has none at all. I called Leading Edge Electric.
  • Window Tinting. The current tinting is old, degraded, and peeling off in places. I called Spotshots Windows. It might be delayed depending on other costs.

Bathroom Work

At the moment, space wise the bathroom is ADA compliant. Currently there are no grab bars so we’ll be adding them in some time during the construction work.

Construction

  • Asbestos Testing. Landlord has someone coming out.
  • Permit for Demolition work.
  • Verify walls are removed and ready for paint.
  • Longmont needs to inspect the changes.

Preparation

  • Need paint and paint gear for the walls.
  • Bring over the two spare sheets of slatwall and mount. See how many more are needed and procure them.
  • For the storage behind the north extension, get some general utility shelves. Customers and gamers won’t see these shelves (generally).
  • Get Ikea Kallex shelf units for the used board games and for terrain in the miniatures gaming space.

Installation

  • Empty the storage shed and get set up in the shop.
  • Empty the storage space in the game shop and get it moved over.
  • Bring over non-retail and easy to carry assets. Posters, pictures, board games, anything else that isn’t needed at the shop.

Utilities and Services

  • Transfer phone number and internet to my LLC.
  • Start Electricity
  • Start Longmont Services (water, trash, sewer).
  • Investigate Security Services including cameras
  • Investigate a Cleaning Service. Carpets, planks, windows, bathroom?
  • Investigate card singles insurance requirements.

Moving Company

Basically moving the shop to the new space. Wrapping up the shelf displays and loading them into the truck. Boxing up wall games and fixtures.

Final Move

  • Notify current lessor that we will not be extending the lease.
  • Box up games and accessories on the slatwall.
  • Remove slatwall and mount in the new shop.
  • Remove security system, camers, cables, and mirror.
  • Remove wall hangings.
  • Remove posters and stickers and clean front windows.
  • Stop electricity
  • Stop phone/internet (change to new space)
  • Stop Longmont services (water, trash, sewer)

Finals to not forget

  • Purchase safe and have installed
  • Review lease for what needs to be done at the old shop
  • Update distributor and publisher addresses (see wiki)
  • Update USPS (mailbox) and turn in key.
  • Update Google address so folks can find us
  • Update business license and deliver to vendors.
  • Notify the IRS
  • Update business cards
  • Get a drink cooler as we can sell refreshments at the new place.
  • Replace the phone
  • Replace the POS as it’s end of life.
  • Build Miniatures tables.
  • Shut down storage unit if empty
  • Need a desk and chair for the office.
  • Find a better chair for the POS area.
  • Notify bookkeeper
  • Replace the 12′ ladder. The current one is owned by James
  • Look into demo tables for games.

Posted in Game Store, Gaming | Tagged , | Leave a comment

Game Store Move: The Lease

Now the fun begins. I passed the lease on to my lawyer and spent the evening reviewing it myself to be familiar with what’s being said. In general most seemed okay however I pushed back on the requirement to provide financial details of my business every year.

I got the updated lease back from my lawyer. He said the phrasing was a bit outdated and several bits needed clarification. I was able to update the definitions for Lease Commencement and Rent Commencement. Here are the pieces my lawyer either excluded or recommended review.

  • There was a statement about future assigned parking spaces, making sure I had at least 2.
  • For the pylon sign (next to the street that has several retail space signs), they had a one time $500 charge. I’d asked if they were creating a sign for that fee since maintenance was included in the triple net fee (external maintenance fee; calculate the costs over the past year, divide by 12, and again by the space I occupy).
  • Some assignment questions (that’s when you sell your business to someone else for example). They wanted me to continue to be liable and to pay them 50% of what I would receive if I sold the business. Both statements were removed.
  • Then if I overstay the term of my lease, a 150% monthly rent which my lawyer wanted reduced to 120% if possible.
  • Then that I was not getting interest on the security deposit. My lawyer said it’s somewhat common but we can ask.

One of the problems right now is the current tenant is waiting on his space to be cleared of kitchen gear so he can move in. Since I’m supposed to take possession Oct 1st, and the current tenant hasn’t moved a thing yet, this weekend should prove interesting. The lease does have a statement that due to unforeseen circumstances, possession might be delayed. I’d sent an email over to ask if there’s a timeline as I need to schedule several work crews to get the space ready.

Posted in Game Store, Gaming | Tagged , | Leave a comment

Game Store Move: Announcement

The following announcement has been posted to social media!

Important announcement. The Atomic Goblin, after being in one place for over 10 years, has exploded in popularity to the point that we absolutely must move into larger digs. Over the next 5 months we’ll be doing construction and preparing the new spot for all of our gaming friends. We’ll have twice as much space for gaming and the ability to bring in more of your favorite games and accessories. We look forward to continuing to see all of our friendly faces at the new location. Expect to see flyers with every purchase at The Goblin providing more details on the move. Stay tuned!

Should be fun!

Posted in Game Store, Gaming | Tagged , | Leave a comment

Game Store Move: Approvals

On the assumption we’d get the new space, I created a 5 page Todo list of things we’d need to do. I had a business dinner with the team last Saturday, bringing folders for all with the todo list, the possible business lines list, the above floor plans plus some alternatives, and a roughed out business plan (I’m going through a class to make it more professional and focused.

The todo list also contained several estimates on getting work done such as an electrician to add more outlets, demolition company to remove the walls, carpet company, and a few more things to think about, cost wise such as painting the walls. I also reached out to the graphics firm I’ve used for flyers and business cards in the past.

I sent the Letter of Intent off to my lawyer for review after I made a few changes to the original one, got updates from him and sent it off to the landlords.

And we got the place! We take possession October 1st and begin work on getting it ready for moving in in January and being 100% the gaming destination February 1st.

Yesterday I received a proposed sign for the new shop. I did add some text in the top windows and the store logo to the front doors. I’m not sure yet if those will be approved so we’ll see.

Here’s the current store front.

And here’s a mock up of the proposed new sign. I passed it along to the landlords and they approve!

Posted in Game Store, Gaming | Tagged , | Leave a comment

Game Store Move: Different Space

Well, I did chat with the landlords for the Hobbydown space, gave them a quick elevator speech. Of course they’re still looking at the nationwide spa taking it however they do feel we are the best choice but first come, first served. If it falls through, we still have a chance.

Unfortunately, no further emails from the first place but the second place called us. The 1517 space we liked is being taken by a nationwide Vet clinic. However they have another place in the same mall that’s becoming available at 1515 Main. We did a walk through last week and I sent an email over to the owners/realtors (same folks) with my requirements. The space is currently a cleaning supply service. Their lease ends end of August and he goes month to month. My lease ends January 31st.

Current floorplan. The front is to the right (or east side) with lines where the double doors and windows are. The back has a pair of lines where the back door is:

The request is to leave the north extension/wall and the vertical wall and remove the perpendicular wall to it. Plus remove the storage walls towards the north wall and against the west wall. As the floor from the vertical wall to the front of the store is concrete, replace the carpet in the back and put down new carpet across the entire floor.

I’d put glass cabinets from the east side of the north extension to the east wall, leave the space behind the wall for storage and the office for, well an office. Miniatures tables on the other side of the vertical wall and card and board gaming tables between the vertical wall and retail space.

We’ll see how this works out. But the new place does have appeal. It doubles both our retail space and the gaming space. Let’s see if they are willing to move forward with the changes.

New floor plan after walls are removed and tables in place.


Posted in Game Store, Gaming | Tagged , | Leave a comment

Game Shop Move

Okay if you recall I bought my Friendly Local Game Store (FLGS) last year. In fact, in a couple of weeks, it’ll be exactly a year. As a reminder, I stumbled onto an ad for a Board Game Cafe up in Saskatoon Canada. Looked interesting and I spoke to my good friend, who was the owner of my FLGS along with his wife. Just to be clear, I do go to many of the surrounding game shops but Atomic Goblin Games is my F LGS. After discussion and months of review, getting financials together, getting a loan, approvals from my wife, I purchased my FLGS, which is now actually my FLGS.

I did hire the former owner as my store manager/purchasing manager. I also kept on the other employees and gave them all raises, now being paid above minimum wage (just a little though, they do get a 30% discount on games). I also brought in the former game store owner where my guy used to work, as a contractor for an occasional couple of hours of work when needed and as a mentor to help me on my way as a retail game store owner. These guys have been great, let me just say. Plus the number 1 ‘senior associate’ is in line to be assistant manager.

Anyway, we’re now bursting at the seams. Lots of gamers, lots of customers, and lots of product. Humorously last year I told the “management team” that I’d want to open a second Atomic Goblin Games Too or move to a bigger location and they were all, “well, get some experience and let’s see in a year or two where we are”.

Since I took over and since I implemented changes, over the course of the year since I purchased it, we’ve had the best 12 months in the almost 11 year history of the store, most of the last 12 months have been the best month in the 11 year history, and right now we’re better than 50% higher in sales than last year at this time.

The problem with where we are is the management companies (yes two) have not been helpful in us expanding into the mainly empty space next door. This would let us expand another 600 or so square feet, from 1,400 to 2,000 square feet mainly (I don’t have an exact number so close enough). It’s being used a storage for the end unit, a gas station/convenience store. Not only that, I’ve been trying to get the lease in my name for over a year, and a couple of places won’t let me take over the old account until the lease is in my name. I can’t even replace the carpet (3 year old carpet and it’s been 10 years) without permission from the owner due to the cost (over $10,000 amount requires approval).

Anyway, enough of that. What’s new?

Well, since I can’t get the lease and can’t expand, and must expand we’ve been checking out possibilities in the city. There are three other game stores. One is in downtown (5th Street) and two are close to each other on the corner of Main Street and 119, another major road that passes through Longmont. The downtown shop is mainly comics with games and puzzles. The two on the corner are either mainly Magic specific or Warhammer specific. We are the board game shop in town.

We had one really good choice. A bit larger than we needed but with some leeway from the owners, and they were very positive, we could grow into the space. It’s a bit over 5,000 square feet which doesn’t count the upstairs that they were throwing in for free. Unfortunately they were working with a nationwide Salon first. But we were lower initial maintenance costs, no need to drill holes in the foundation for water drainage and just needed the bathrooms upgrade and replace the carpet (which the salon would need anyway). It’s also a pretty good price in general, $12 sq/ft NNN ($7.50 sq/ft) (NNN is basically the exterior maintenance costs; parking lot, grass, trees, roof, etc).

It was the spot Hobbytown occupied so was already a gaming/hobby destination in town. Win-Win, but unfortunately again, not really available. I did send a couple of emails asking if there was any way we could slide in but in the event we couldn’t, we would still look around.

Link to the listing if you’re interested: https://www.coloradogroup.com/property/1935-main-street-suite-b-longmont-co/ (it may be gone in the future of course 🙂 ).

Here’s the floor plan. The lower piece they were throwing in for free as it’s inaccessible otherwise.

And here’s one of the pictures.

The nice thing is it is on Main Street, there’s a BBQ Place in the end unit, Dairy Queen and Wendy’s next door. Pretty good for the gamers and such coming into the shop.

Anyway, my wife and I went driving around town today to check out places based on several commercial listings we found on line. One is a former Big 5 Sports big box store. 10,000 square feet. We checked it out and it’s really 5,000 of retail and a second floor with an additional 5,000 of office space. Could be interesting however it’s really quite large but I will call to check.

There were several other possible places. A couple in a kind of skeevy strip mall. Not horrible in general, there are several restaurants and fast food places (like Five Guys and Chipotle’s) but I dunno, kind of an off-putting feel. Plus there were 4 open spaces, not lending itself to confidence in the customers.

A second block was better, a former K-Mart converted into several shops like a Big Lots, a Cricket, and an Arc shop. It looked good looking into the window, but when I checked on line, it’s almost 8,000 square feet.

We went into one strip mall and from the listing, it looked like the open space was taken. We continued on but my wife said there was another open space that wasn’t in the listing. We were able to locate the commercial listing on another site.

Listing here: https://www.loopnet.com/viewer/pdf?…Bridge%20Park%20Plaza15151517%20Main%20St.pdf

In the picture, it’s 1517 units A and B.

It’s a touch over 3,000 square feet which fits nicely into our expansion needs. Unit A is separated from Unit B by a wall with a double wide door like opening (it was a single use). The front part is 1,500 square feet, the back the same. With us using the front part as retail, we could use Unit B as the gaming area. More room for tables and such. Plus if we are able, we might pick up Unit C and add it as an RPG gaming area with two walls as a corridor so RPGers could have quiet to play.

The cool thing? There’s a sliding garage type door. The gap in the wall between Unit B and Unit C. Now that would be great during nicer days. Open the door for the gamers and expose them to fresh air.

The bottom of the pic faces Main Street. If you clicked the link, there’s a Taco place, Pizza Hut, and an ice cream place. It feels so much nicer, neighborhood wise with outside seating at the Taco place. It just feels like a good place to move to. At 3,000 square feet (or more if we take Unit C or even Units D and E (the office space)), it would let us grow just a little better and saner than the 5,600 or 10,000 square feet places and in 5 years, maybe then move into a much larger space.

Cool beans.

Looking forward to seeing how this shakes out.

Posted in Game Store, Gaming | Tagged , | Leave a comment

Pan Fried Rainbow Trout

Basically I do this when I find a recipe on line that I like and want to make sure I can find it again 🙂

Ingredients

  • 2 Rainbow Trout fillets, boned and with skin.
  • 1/2 teaspoon kosher salt
  • 1/4 teaspoon black pepper
  • 1/4 teaspoon garlic granules
  • 1/2 cup butter (half a stick)
  • 1/2 tablespoon fresh lemon juice
  • 2 tablespoons minced parsley

Instructions

  • Rinse off the trout and pat dry with a papertowel.
  • Sprinkle the skin side with half of the mixture of the salt and pepper. I find doing it as a pinch makes sure it’s spread evenly.
  • Heat the butter in a 12 inch or so nonstick skillet. When the butter starts bubbling, swirl it around to make sure it’s spread evenly around the pan.
  • Add the two fillets skin side down and sprinkle the rest of the salt and pepper mixture on the fleshy side. Press on the fish to make sure the skin touches the pan.
  • Cook time is about 3 minutes.
  • When the skin side is done, sprinkle the garlic on the fillets and flip it over. Cook for about 3 minutes.
  • After removing the fish, put the lemon in the pan and mix it up. Then drizzle the sauce over the fish.
  • Sprinkle with parsley and serve.
Posted in Cooking | Tagged , , | Leave a comment

Quick and Dirty Kubernetes

Overview

At times you want to quickly throw up a Kubernetes cluster for some quick test or another. While I do have several Kubernetes clusters on my Homelab once in a while you want to do a quick test or even follow a tutorial to get familiar with this or that tool.

This time I’m using a tool called kind. See the References below to find the link to the site.

Installation

First off you’ll need to have a docker server in order to install the various tools. Next you’ll have to install the kind tool.

# go install sigs.k8s.io/kind@v0.17.0
go: downloading sigs.k8s.io/kind v0.17.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/spf13/cobra v1.4.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/alessio/shellescape v1.4.1
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c
go: downloading github.com/pelletier/go-toml v1.9.4
go: downloading github.com/BurntSushi/toml v1.0.0
go: downloading github.com/evanphx/json-patch/v5 v5.6.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading sigs.k8s.io/yaml v1.3.0
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading github.com/google/safetext v0.0.0-20220905092116-b49f7bc46da2

Once kind is installed, start up your test cluster. Note that kind is installed off your home directory in the go/bin directory. Either add it to your path or add it to the command.

# go/bin/kind create cluster --name nginx-ingress --image kindest/node:v1.23.5
Creating cluster "nginx-ingress" ...
 ✓ Ensuring node image (kindest/node:v1.23.5) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-nginx-ingress"
You can now use your cluster with:

kubectl cluster-info --context kind-nginx-ingress

Thanks for using kind! 😊

See if the cluster is up.

# kubectl get nodes
NAME                          STATUS   ROLES                  AGE     VERSION
nginx-ingress-control-plane   Ready    control-plane,master   3m21s   v1.23.5

And it’s ready to be used. Pretty interesting.

References

  • https://kind.sigs.k8s.io/
Posted in Computers, Kubernetes | Tagged , | Leave a comment