Docker Best Practices

Posted on December 22, 2021 by Freejack

Use the Official Docker Image for your Base Image. Download the image from the docker.io site.

Never use the Latest tag for an image. In order for consistency, identify the version you want to use. This also prevents potentially breaking changes.

Use the image that satisfies your requirements. Many time something like Alpine, which is a tiny image, has all the functionality you need for your image. Something like Ubuntu/Debian or CentOS will have a ton of extra, unnecessary tools. Similar to when we build server, only start services that are needed and disable or even remove the services that aren’t used.

Optimize Caching Image Layers. If you check the dockerfile for an image, you can see how what’s been installed. Each command adds a layer to the final container. To optimize, consider how your container is built. Whatever is changed the most often, should be later in the final container. For example, your application code might be best at the end of the dockerfile. Then the rest of the unchanged layers won’t need to be rebuilt as everything after a rebuilt image is also rebuilt.

docker history image:tag

Exclude unnecessary content to reduce the size of the image. Use the .dockerignore file. Ignoring .git, etc.

Remove unnecessary files after the container is built. Use multi-stage builds. Lets you use staging images to prevent development files from the final image. For example, when you compile a C program, you have makefiles and .obj files. You would use a multi-stage build where the first image compiles the final program but the last image only contains the compiled binary.

Set up an appropriate user to run the final application and not root. It’s a security bad practice.

# create group and user
RUN groupadd -r tom && useradd -g tom tom

# set ownership and permissions
RUN chown -R tom:tom /app

# switch to user
USER tom

CMD node index.js

Scan image for vulnerabilities. Use the docker scan from the docker hub system. docker scan image:tag.

Posted in Computers, Docker | Leave a comment

Replacing An OKD Master

Posted on November 28, 2021 by Freejack

I’m running an OKD4 (aka an upstream Red Hat OpenShift Container Platform v4) cluster at home. Recently my bldr0cuomokdmst1 master node crapped out. I couldn’t even log in to determine the problem. For my current Kubernetes clusters, I’m casting the logs to my ELK clusters but hadn’t done that for the OKD4 cluster yet. So no idea why it failed. Now I need to delete the old master, clear out the configuration, and add it back in again.

Preparation

First off, you should have already replaced the certificate in your installation process. See my OKD4 Installation post for details on how to do that. As a reminder, the certificate is only good for 24 hours for a new cluster build. After that you need to retrieve the cluster certificate and add it to the ignition file.

High Availability Proxy (haproxy)

I removed the failed master node from the cluster in part because it was causing timeout problems with managing the cluster while I was researching the solution. Sometimes the console would work, other times I suspect it was trying to get to the failed master and timing out. However deleting the master and adding it back in is a pretty quick process so removing it from the haproxy configuration might not be necessary.

Log in to your cluster haproxy server (bldr0cuomokdhap1) and in /etc/haproxy, edit the haproxy.cfg file to comment out the bldr0cuomokdmst1 server entry from the configuration. Don’t delete it as we’ll be uncommenting it when the master has been recovered.

Remove Master

Drain the node from the cluster. While the pods won’t be removed, the node will be cleared from the system so it’s not accepting incoming connections any more.

$ oc drain bldr0cuomokdmst1 \
  --delete-emptydir-data \
  --ignore-daemonsets \
  --force

This will take some time as requests to the failed master will need to time out. Now verify bldr0cuomokdmst1 has been removed as an endpoint in the cluster. The IP is 192.168.101.100 which should be missing from the output below.

$ oc describe svc kubernetes -n default 
Name:              kubernetes 
Namespace:         default 
Labels:            component=apiserver
                   provider=kubernetes 
Annotations:       <none> 
Selector:          <none> 
Type:              ClusterIP 
IP Families:       <none> 
IP:                172.30.0.1 
IPs:               172.30.0.1 
Port:              https  443/TCP 
TargetPort:        6443/TCP 
Endpoints:         192.168.101.102:6443,192.168.101.103:6443 
Session Affinity:  None 
Events:            <none>

Finally delete bldr0cuomokdmst1 from the cluster.

$ oc delete node bldr0cuomokdmst1
node "bldr0cuomokdmst1" deleted

Clear etcd

Since etcd is on the masters and not a separate cluster, we’ll need to remove the etcd configuration as well. You’ll log in to a working etcd pod, remove bldr0cuomokdmst1, then remove any secrets that belong to the master node from the cluster.

Verify Status

$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'
2 of 3 members are available, bldr0cuomokdmst1 is unhealthy

Working Pods

$ oc get pods -n openshift-etcd | grep -v etcd-quorum-guard | grep etcd etcd-bldr0cuomokdmst2                3/3     Running     0          33d etcd-bldr0cuomokdmst3                3/3     Running     0          33d

Clear Configuration

In this step, you’ll log in to one of the above working pods (2 or 3) and remove bldr0cuomokdmst1 from the etcd configuration.

$ oc rsh -n openshift-etcd etcd-bldr0cuomokdmst3
Defaulting container name to etcdctl.
Use 'oc describe pod/etcd-bldr0cuomokdmst3 -n openshift-etcd' to see all of the containers in this pod.

Get the member list

# etcdctl member list -w table
+------------------+---------+------------------+------------------------------+------------------------------+------------+
|        ID        | STATUS  |       NAME       |          PEER ADDRS          |         CLIENT ADDRS         | IS LEARNER |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
|  e43c9f92fda4af5 | started | bldr0cuomokdmst3 | https://192.168.101.103:2380 | https://192.168.101.103:2379 |      false |
| ac4ca03e8d200e17 | started | bldr0cuomokdmst2 | https://192.168.101.102:2380 | https://192.168.101.102:2379 |      false |
| c7804a193b578f80 | started | bldr0cuomokdmst1 | https://192.168.101.101:2380 | https://192.168.101.101:2379 |      false |
+------------------+---------+------------------+------------------------------+------------------------------+------------+

You can see that bldr0cuomokdmst1 is in the configuration. Now remove it.

# etcdctl member remove c7804a193b578f80
Member c7804a193b578f80 removed from cluster 617aed10ec5206e3

Finished. You can exit the etcd pod now.

Remove Secrets

There are three secrets for each master node. You’ll need to get the list and then remove them from the cluster.

$ oc get secrets -n openshift-etcd | grep bldr0cuomokdmst1
etcd-peer-bldr0cuomokdmst1              kubernetes.io/tls                     2      205d
etcd-serving-bldr0cuomokdmst1           kubernetes.io/tls                     2      205d
etcd-serving-metrics-bldr0cuomokdmst1   kubernetes.io/tls                     2      205d
$ oc delete secret -n openshift-etcd etcd-peer-bldr0cuomokdmst1 secret
"etcd-peer-bldr0cuomokdmst1" deleted
$ oc delete secret -n openshift-etcd etcd-serving-bldr0cuomokdmst1
secret "etcd-serving-bldr0cuomokdmst1" deleted 
$ oc delete secret -n openshift-etcd etcd-serving-metrics-bldr0cuomokdmst1
secret "etcd-serving-metrics-bldr0cuomokdmst1" deleted

And the failed bldr0cuomokdmst1 server has been completely removed from the cluster.

Rebuild Master

This process follows the initial build process except with a single node. You’ll boot the server to an ISO image, update the boot line, approve the Certificate Signing Request (csr), and monitor the node.

haproxy Node

If you cleared the master server from haproxy, you’ll need to uncomment that line in the haproxy.cfg file plus you’ll need the boot node so that’ll also need to be uncommented. Log in to bldr0cuomokdhap1 and in /etc/haproxy edit the haproxy.cfg file. After updated, restart haproxy.

Start Guest Image

Update the boot settings for the image to boot to the BIOS after started. Once started, attach the Fedora CoreOS ISO, make sure the system boots to CD-ROM, and save and start the image.

In the Live Image, tab to the boot line and enter in the following parameters at the end.

coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.101.100:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.101.100:8080/okd4/master.ign

Monitor the console and you’ll see the system start up, then start downloading the image, and it’ll boot. Remove the ISO from the image after it starts and reboot the system. Initially it’ll retrieve the newest image so you may see it reboot again.

Approve CSRs

Now you need to review and approve any outstanding CSRs.

$ oc get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-k258q   11m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

There should really only be one but if more than one, feel free to investigate. Once done, approve the outstanding CSR and pods will start on the bldr0cuomokdmst1 node.

$ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/csr-k258q approved

At this point, the replacement master node is part of the cluster and is creating pods. As there are quite a few (43 at my last count) on a single master node and I’m on high speed WiFi, it will take several minutes.

Verify etcd Configuration

Once all the pods have started and the new cluster member is Ready, verify etcd is also working. First off, check the health of the cluster.

$ oc get etcd -o=jsonpath='{range .items[0].status.conditions[?(@.type=="EtcdMembersAvailable")]}{.message}{"\n"}'
3 members are available

That looks good. Log in to the same etcd pod you did at the start and check the table output.

$ oc rsh -n openshift-etcd etcd-bldr0cuomokdmst3
Defaulting container name to etcdctl.
Use 'oc describe pod/etcd-bldr0cuomokdmst3 -n openshift-etcd' to see all of the containers in this pod
# etcdctl member list -w table
+------------------+---------+------------------+------------------------------+------------------------------+------------+
|        ID        | STATUS  |       NAME       |          PEER ADDRS          |         CLIENT ADDRS         | IS LEARNER |
+------------------+---------+------------------+------------------------------+------------------------------+------------+
|  e43c9f92fda4af5 | started | bldr0cuomokdmst3 | https://192.168.101.103:2380 | https://192.168.101.103:2379 |      false |
| 630bbe550c81b877 | started | bldr0cuomokdmst1 | https://192.168.101.101:2380 | https://192.168.101.101:2379 |      false |
| ac4ca03e8d200e17 | started | bldr0cuomokdmst2 | https://192.168.101.102:2380 | https://192.168.101.102:2379 |      false |
+------------------+---------+------------------+------------------------------+------------------------------+------------+

haproxy Update

The final task is to remove the boot server from the haproxy configuration. Log in to bldr0cuomokdhap1 and edit /etc/haproxy/haproxy.cfg. Comment out the boot server lines, save, and restart haproxy.

References

Posted in Computers, OpenShift | Tagged , , | Leave a comment

Original Car Wars Play

Posted on October 30, 2021 by Freejack

Morning! Jeanne and I played the old Car Wars Saturday night. Took a bit to track down some of the bits and there are a few others that I haven’t found yet but I’m still hunting.

Top pic is my chit carrying case. Under are the bags of chits from the game and various expansions.

Top pic is the vehicle record sheet I created back then. I was a graphics artist so that’s an ink and paper creation. Under pic is the same record sheet but after spending about a half hour filling it out. I used one of the sample cars but still had to track down information so we had damage points, ammo, etc.

Amazingly I found my first arena, the top pic. All the post-its are holding chits down where they were. Apparently we were playing a game who knows how long ago and intended on continuing. Clearly not. The lower pic is after all the post-its have been removed leaving the original damage and car in place. I suspect the other vehicle dropped off sometime in the past and it may be in a box somewhere.

To the game itself. Jeanne and I have the same car and I didn’t want to add extra bits to the game so no dropped debris from hits. In the pic you can see the original arena, our vehicles, flaming oil chits, and the official turning key. We both have linked rocket launchers with a targeting laser (+1). To hit is 8 or higher on two dice by default and a 7 with the laser and 2d6 damage. To the rear is a Flaming Oil Dispenser, 3d6 damage. We’re coming in at 15 and 20 mph, both with a handling class of 3 and acceleration of 5mph. We also have the Control Table and Speed Table close at hand.

In the first pic, after doing some maneuvering, Jeanne cut a hard left in front of me just missing hitting each other (cars not chit dimension). In the next pic however, she learned and dropped a flaming oil patch in front of me that I can’t avoid! 11pts of damage to my undercarrage!

In the final pic, you see the ending of the game. She decided to head on crash and the damage gave her 2 pts left of front armor. I fired my rockets into her front end (and took damage too, 2″ radius explosive damage). I still had armor after the explosion but I breached her armor, removed both her rocket launchers, her engine, and her targeting laser leaving her to scramble out of the wreckage and hotfoot it over to the exit.

In comparing the two games, Jeanne liked playing in both. The issues with the game with chits is they were small and tended to adhere to the fingers so moving them around was a bit annoying at times, something 6th fixed with miniatures. We liked the original turning key better than the new turning key that slides under the base of the miniatures. Certainly set up was significantly quicker with 6th and having cards for the various weapons and accessories. If we had pre-generated vehicle record sheets, setup would have been about as quick I guess. Having to hunt through the book for stats took quite a bit of time and I had to look up the Difficulty values for some things like taking damage but once known, it was easy enough to manage (taking damage: 1-5 D1, 6-9 D2, 10+ D3). We really liked the MPH for speed vs the 1-5 increments. Made it a bit more realistic feeling.

Overall 6th is better just because of the miniatures and having cards for the weapons and accessories. 1st-4th is better though if part of your joy is building the best car. Think of it as closer to a Collectible Card Game like MtG where you’re creating the “Best” deck. And the earlier game has a ton of adventures. But we like both and Jeanne is looking forward to breaking out either game.

Posted in Gaming | Tagged , | Leave a comment

Car Wars and Car Wars 6E, A Comparison

Posted on October 14, 2021 by Freejack

I played Car Wars back with the pocket edition and a bunch (not quite all) of the expansions. I even ran a AADA chapter in Virginia for a bit and when we got into a car accident one Sunday afternoon, the idiots who started the problem pointed to my AADA Autodueling bumper sticker to prove I was driving up and down I-95 causing trouble (with my wife, BIL, and two kids in the car of course).

For those who haven’t played the original Car Wars, it’s basically a Mad Max, Zelazny’s Why Johnny Can’t Speed, junkyard car battles, and Robot Wars mixed together.

Car Wars 1st Edition

Car Wars is based on a dystopian future with shortages and minimal government safety in the badlands. Much like junkyard battles with the addition of weapons like machine guns, rocket launchers, and recoilless rifles.

Basically you select a pregenerated car which has a chassis, body, wheels, engine, armor, accessories, and weapons (and a crew) and take the car to an arena and battle it out. Winner takes all.

To keep things fair, you set a price limit. For arena combat, you have ‘Division’ level combats. A Division level 10 area might mean you’re limited to a car that’s worth $10,000. There might be Division 12, 15, 20, 25, and even Unlimited arenas.

This also means you can build your own car (like Robot Wars). There are rules on how to build a car in the original Car Wars. To me, this was half the fun. My first ever computer program back in the early 80’s was a Car Wars Vehicle Generation Program. I recently found the source code and it’s up on my github site now 🙂 You had several tables with the things you need to build a vehicle. Chassis, body type, engine, etc.

For movement, there was a movement key. An ‘L’ shaped key with Difficulty (D) markings. D1 – D6. The sharper the turn, the more D you accumulated. This was compared with your speed to a table. The lower the speed, the less likely you’d crash. As you went faster, control was more easily lost when making maneuvers to the point that you needed to roll a die to see if you lost control. If you did, or if you exceeded your control value by too much, you went to the crash tables. You could skid, slide, or even roll the car over, one side for every 10mph you were going.

One of the very cool maneuvers was a Bootlegger Reverse. You’d use the key to turn your car sideways and then point back in the opposite direction at 0mph. Cool stuff.

Combat was basically you rolled two dice based on your weapon and getting under a specific target number means you hit the target. However there were a ton of modifiers. Distance, speed, cover, etc would increase or decrease the target number. If you finally hit, the target would take damage generally to the armor first, then to components or even crew killing the driver and/or gunner.

There were several rules on collisions, running into or over obstacles, pedestrians, plus lots of new vehicles over time like motorcycles, trailers, helicopters, boats, and even tanks.

Subsequent Editions

I didn’t continue with the game after the 1st edition. From what I can discover, 2nd, 3rd, and 4th editions were pretty similar if pared down. 1st had a ton of expansions and gear. Arenas, road sections, adventures, even books on background of various areas of the US and world. Subsequent editions just improved the rules but you could still use the old expansions.

5th Edition

5th Edition make some significant changes. You had a Division booklet with two pregenerated vehicles and fought each other. I looked at it back then and my preference was to create the vehicle, not use a pregenerated one regardless of who created it.

6th Edition Kickstarter

When the 6th Edition kickstarter was announced, I was skeptical. But in reviewing the information, it seemed like it was returning to the roots of the game to some extent. And it had vehicle miniatures. I put up my money and got everything I could. I like Car Wars and if it’s going back to where I can build vehicles, I’m in.

We received the first package yesterday morning and played last night. The first box is the Double Ace box.

  • Core game with two boxes of miniatures (red/yellow and blue/green sets) and a rules box with cards, 5 set of dice, chits, and pregenerated cars.
  • Miniatures Set 1 with 5 more miniatures
  • Miniatures Set 3 with 5 more miniatures
  • Dice Bag
  • Road Tiles with more chits (Control, Ace, and Damage)
  • Crew Pack with more crew options and weapons
  • Dropped Weapons pack with mines, paint, spikes and other debris you can drop from your vehicle
  • Linked Weapons Pack with 8 linked weapons and turreted weapons
  • Armory Pack with 8 more weapons like Gauss and Railgun and turreted weapons

We basically played just the core set last night. My wife playing green and me in blue.

The core game has a dashboard now. It has armor tracks for all four sides of the vehicle, a tire track, engine track, and speed track. Tires and Engine start at 10 points. Then you select a Build Point target for the game. On the tire track, there are control markers. You get a control chit for each visible marker on the dashboard. As you lose points, you get fewer control chits at the start of your turn. Four is max for tires plus if you’re going slower, you can pick up a control chit for a max of five. In addition your speeds are 0-5 and you can lost max speed depending on tire damage down to 1 max speed.

You have a set of cards for your vehicle. It has accessories, upgrades, weapons, and crew cards which also have accessories and weapons.

The vehicle cards have build point values in blue in the upper right of each card, from 0 to 8. And damage values in the lower right. Crew cards have crew points in red, from 0 to 8, in the upper right and damage values as well.

We selected 16 Build Points (BP) per the example in the rules, a small car setup. In addition to 4 Armor Points (AP) and 4 Crew Points (CP). You have to have two crew. Rookies are worth 0 so you can get two rookies; driver and gunner, or better crew with abilities.

There are 5 sets of dice. A yellow set, green, red, black, and white set. The dice have three types of faces. Damage (a star), shields, and mechanical. It can be 1 or 2 items (like 2 stars).

For movement, the turn key has five Difficulties or D maneuvers. D0 through D6. D1 is a green die, D2 is yellow, D3 is red, and d6 is white. For weapons the cards show how many of each die you roll for damage. Two yellow, 1 red, and 1 black (for example).

In maneuvering, if you turn, you roll your speed number of yellow dice (so if speed 3, you roll 3 yellow dice) plus the die for each of the D for the maneuver you’re attempting. For example, if you make a D3 move (45 degree turn) and are at speed 3, you roll 3 yellow dice for speed plus an additional green, yellow, and red die. For every shield face that results, you lose one control chit (remember you start with 4 or 5 chits each turn). If you run out of control chits and still have shields, you lose control.

For combat, I roll the dice as indicated on the weapon plus any additional as listed by equipment. The number of stars is the damage taken. If a mechanic face shows, if the weapon has extra abilities like explosion, fire, or tire damage, that’s taken. The defensive player rolls the number of yellow dice equal to their speed plus any additional dice based on upgrades if any. Any shields block the same number of damage points.

For every two car lengths, the defensive player can reroll their dice and if they have any Ace tokens, they can spend them to reroll any single die.

Finally

The original Car Wars had a lot of tables which added complexity to the game. What’s your Handling Class, speed, what difficulty was the move, look things up, okay roll the dice and check the crash table depending on your vehicle.

The 6th Edition has replaced much of that with chits and dice. It does speed up the game and my wife, who’s not really into the more complex table lookup type games, actually enjoyed the quick game last night.

Costs

The price will be the stopping point for all but the most fervent Car Wars fans. At $310.00 it might be just a bit out of reach. 🙂 You can get the starter sets without all the extra bits for much less though. For us fans though, I think this is a good upgrade to the original game. Especially if my wife enjoyed the game. That makes all the difference. I can’t wait to get more games going with her. I’m considering running a few games at my FLGS just to see if there’s interest. Maybe Jamie will stock a game.

Kickstarter vs Retail

Okay, I asked a question on the forums about one of my purchases on the and have compiled a list of what the Kickstarter stuff costs and what the retail cost is.

Site: https://carwars.sjgames.com/products/

Kickstarter/Retail Price:

  • Double Ace – $200/375 (bought separately from the store, below items are 442.55)
    • Core Box – 75/149.95
    • Rules Box – /30
    • Miniatures Box A – /60
    • Miniatures Box B – /60
    • Miniatures Set 1 – 40/59.95
    • Miniatures Set 3 – 40/59.95
    • Dropped Weapons Pack – 20/29.95
    • Crew Pack – 15/29.95
    • Armory Pack – 10/24.95
    • Linked Weapons Pack – 15/24.95
    • Road Tiles – 30/49.95
    • Dice Bag – 10/12.95
  • Double Ace Upgrade Bundle – 165/ (bought separately from the store, below items are 199.80)
    • Miniatures Set 2 – /59.95
    • Miniatures Set 4 – /59.95
    • Vehicle Guide – Not available in Store
    • Wrecks – Not available in Store
    • Playmat #2 – /59.95
    • Dice Pack – /19.95
  • Two Player Starter Set – Red/Yellow – 70/79.95
  • Two Player Starter set – Blue/Green – 70/79.95
  • Uncle Al’s Upgrade Pack – 15/29.95
  • Playmat #1 – 45/59.95
  • Playmat #3 – 45/59.95

Not part of kickstarter (as far as I know of course):

  • Uncle Al’s Arena Supplies #1 – /10
  • Car Wars Clear Bases – /12.95
  • Car Wars Colored Bases – /12.95

Totals:

  • Kickstarter: $610.00
  • Shop: $1,035.50

So I saved $425.50 by going All-In on the Kickstarter. Which only helps if I actually get folks playing it. 🙂

Posted in Gaming | Tagged , | Leave a comment

Recommended Games

Posted on August 27, 2021 by Freejack

Jeanne and I have played several games in the library (see prior post). I did a quick walk through for someone who asked for a list of good games. While there are quite a few games that are pretty good, for the walk through, I did a quick selection vs an indepth review of all the games. So here’s the list I provided.

Ace of Aces You’re flying WWI airplanes. Your opponent has a book with pictures of you and you have pictures of their plane. You select a maneuver which has a page number, go to an intermediate page, then the final page to see what your opponent is doing (including shooting at you).

B-Movie Card Games You’re creating a B-Movie. There are 8 or 10 different decks. You lay down a location, characters, and accessories like a whip. Then your opponents throw monsters at your movie to prevent you from creating the best B-Movie.

Bunny Kingdom Very little conflict. Gridded and numbered board. You pick cards and place your bunnies. There are a couple of squatter cards hoping the official card doesn’t come up but mostly there isn’t a way to take over a space that has a bunny on it.

Castles of Burgundy Manage and increase your hold by rolling dice and selecting resources in order to add farm animals and buildings. This is a very well balanced game for 2 players, 3 players, or 4 players.

Cosmic Encounters Take over alien races with skill, no dice rolling. However you’re playing races that modify the core rules. I think this might have influenced the creation of Magic: The Gathering. Core rules then an alien power that changes a rule which makes the game different every time you play.

DC Deck Building Basically drawing cards based on the DC comic universe from the displayed cards using your current cards to ‘take over’ the displayed card. I find this much better than Legendary BTW.

Discoveries of Lewis and Clark Explore the west and gather routes. Gather native tribes to help in gathering routes.

The Doom That Came To Atlantic City Basically reverse Monopoly. The board is set up with two houses on a similar Monopoly style board. You play an old god and create portals by destroying houses. The person with 6 portals to the nether realms wins.

Elder Sign I like all three of the Cthulhu type games (Arkham Horror and Eldritch Horror) but this is the quicker of the three to play. In this one you’re at a portal and trying to block the coming of the elder gods. Arkham Horror takes like 6 hours to play, Eldritch Horror about 3 hours, and Elder Sign about 90 minutes. There is a second edition Arkham Horror we haven’t tried yet so maybe it’s more streamlined.

Epic Tiny There are several different games like dinosaurs, space, strategy, etc. We’ve played the space one several times. Quick and easy to set up and quick to understand. We enjoy these.

Everdell You have workers and are placing them on locations, events, and such to increase your new location.

Five Tribes You place a grid of tiles and then three meeples on each tile. You pick up the three (more or less), drop one on each tile and the last one has to match the color of a meeple on the last tile. Then you collect whatever the color describes.

Formula D This one is cool. You have multiple different dice from a 4 sided up to a 30 sided, each based on a speed of your car (slower is 4, fastest is 30). You have to slow down for corners so you have to drop your speed or you could spin out and end up in the bushes. Pretty fun game.

Gizmos You’re building engines using marbles from a central pot. Whoever has the best engine at the end of the game, wins.

Gloom You’re trying to kill off your ‘family’ by telling horrible stories about their lives based on the cards drawn. These are clear cards with text and pictures so your negative points can be blocked if someone places a positive card over it. Pretty good storytelling type game.

Horrified Actually a game much easier for younger kids. You’re playing one or two classic monsters; invisible man, wolfman, mummy, etc and the villagers are trying to stop you.

Mountains of Madness You’re flying a plane into Antarctica and to the Mountains of Madness. You’ll flip cards and deal with the results.

New York 1901 You’re creating buildings. Over time you can replace your bronze tiles with silver and gold to increase from a small building to a sky scraper.

The Others This is based on the 7 deadly sins. You are trying to defeat the core monster of each of the sins. Lots of setup but it can be fun.

Pandemic Mainly Iberia although Pandemic itself is pretty good with the expansion.

Photosynthesis You are building trees. The sun rotates around the board so you only get points if your trees aren’t blocked by other, taller trees.

The Red Dragon Inn Card game where four folks have returned from an adventure and are in a tavern enjoying the spoils, drinking, and playing cards. The last one to pass out, wins 🙂

Resident Evil This is a card game similar to DC Deck Building or Legendary and does a pretty good job matching the Resident Evil video game.

Robo Rally Kind of computerized where you’re creating paths using cards to move your robot around.

Splendor A pretty simple game. You have three rows of cards. Raw gems, polished gems, and final gems (or jewelry, I forget 🙂 ). You’re building up your cards and gems in order to buy the next and final row of cards. 15 points wins the game.

Star Wars X-Wing A miniatures game but with very large models for some of the games. You have a wheel for movement and at the end of each turn, you could be destroyed.

Talisman You have three realms. An outer, inner, and the final center task to become the ruler of all the realms. You travel around the board (or expansions, I like the City one but the space one is pretty interesting), pick up sufficient gear to beat other players to the center. The main problem here is folks tend to get a lot more than they really need to win so the final encounter is pretty much a done deal. Still a fun game.

The Thing One of several “one of the team is the bad guy” sort of games 🙂 You’re exploring, trying to discover who the bad guy is. If the bad guy escapes on the helicopter, you lose.

Ticket To Ride Mainly Rails and Sails. Train placement to complete routes. The more routes completed, the more points. Rails and Sails lets you place ship routes too on a world map or map of the Great Lakes.

Trains Another deck building type game except you’re building train routes.

Tzolk’in One of the multiple paths to victory type games. You have several wheels that move during the game.

Wings of War WWI air combat. Like X-Wing, you have plans that move around a board. You select your maneuver (three cards) and flip them. At the end, you might get shot down..

Wingspan This is a pretty cool game for the bird cards. Great pictures. You’re trying to attract birds to your location. You get food, eggs, and birds. Most points wins.

Zombicide A zombie game. Lots of setup and lots of expansions. You’re trying to get from point a to point b, collecting keys or whatnot in order to pass into the prison (for example) and defeating zombies. My wife did an excellent job escaping zombies and bringing out the other players where I was at the exit and ready to just abandon everyone. She’s great 😀


Posted in Gaming | Leave a comment

Mid Year Review

Posted on August 11, 2021 by Freejack

This is more of a redo of the game room into a game library type post. I generally do a COMC as an end of year review so none of the extra information that’s typically added.

In this case the old game room had the Kallax shelves against the walls leaving space in the middle of the room for a table and chairs so my wife and I could play a game now and then. Last year though we bought a really big dining room table (8 seater) and we’ve been gaming up in the dining room instead of the game room.

Last week I whipped up a room layout with the existing Kallax shelves so I could move them around on the drawing and see if things would fit in the way I desired. I currently have 4 5×5 shelves, 2 4×4 shelves, 3 2×4 shelves, 2 2×2 shelves and 5 1×4 shelves.

With the change to a game library vs a game room, I was able to add two more 5×5 shelves which gives me a ton more space plus I have room to add two more 1×4 shelves on top of the 5×5 shelves and another 4×4+2×4 combo.

The process was to build the new 5×5 shelves and put them into the hallway. Then move the games from the existing shelves into the hall shelves. I had to pull down the 1×4 shelves from on top of the 5×5 shelves into the hallway as well before we could reposition the 5×5 and 4×4 shelves. After the existing shelves were rearranged the games were put back into the shelves where ever they’d fit. Then move the new shelves into the room (the white ones in the back). Finally move the second 4×4 shelf to the left side. It holds the mini games and card games plus some display stuff on top.

My wife was a great help in moving things and had a few suggestions for the move and for the final layout. I originally had the two 4×4 shelves at the front of the room but because of the angle of the wall, some of the games would be inaccessible and the door wouldn’t be able to be fully opened. We moved the single 4×4 back enough to let the door shut and put the second 4×4 against the wall as you can see.

The pictures progress from the hallway, a view of the first shelf. Then the side wall shelf followed by the shelves all the way to the back. Then the last shelves to the left. I also was able to put up a few of my posters. I have tons more but I’m a Shadowrun geek so it’s Shadowrun 🙂

My wife’s main question, “are the new shelves enough space to hold the next 5 years of games?” 😀

My final intention for this year’s End of Year Review is a video review discussing my 50 years of gaming, pointing out specific games, and generally providing a bit more targeted information.

And all the pictures are over here if you want to see larger versions.

Gameroom Pictures
Posted in Gaming | Leave a comment

Sequential Updates

Posted on June 15, 2021 by Freejack

Say you want to upgrade all the servers. Straight yum upgrade or apt update.

ansible \
  --inventory/-i [inventory] \
  [pattern] \
  --module-name/-m [command] \
  --args/-a "[module arguments]" \
  --become/-b

For example, in my inventory, I have servers grouped by OS. If I wanted to patch all the CentOS boxes sequentially, I’d run:

ansible -i inventory centos -m yum -a "name=* state=latest" -b

For a Debian server:

ansible -i inventory debian -m apt -a "upgrade=yes update_cache=yes" -b

This then upgrades all servers with that pattern sequentially. One after another. Once done, you can then reboot the servers.

ansible -i inventory centos --args "reboot" -b
Posted in Computers | Tagged , , , , | Leave a comment

Homelab 2021

Posted on June 2, 2021 by Freejack

It’s that time again. It’s a year later and there have been quite a few changes in the Homelab. With the release of VMware 7, I needed to replace my servers with something a bit more current. I repurposed one of the R710’s as a KVM server and replaced the others with Dell R720XD servers. As a note, I’m less a hardware/network guy and more an OS and now Kubernetes guy.

Network

  • High-Speed WiFi connection to the Internet (the black box on the bottom left).
  • Linksys EA9500 Wifi hub for the house.
  • Linksys RE9000 Wifi extender upstairs.
  • HP V1910-48G JE009A 48 Port Gigabit Ethernet Switch.
  • Netgear GS108E 8 Port Gigabit Switch.
  • Netgear GS116E 16 Port Gigabit Switch.

Servers

  • Nikodemus: Dell R710, 2 Intel 5680’s, 144G Ram, 14 TB Raid 5.
  • Slash: Dell R720XD, 2 Intel E5-2697 v2’s, 384G Ram, 27 TB Raid 5 w/Hot Spare.
  • Monkey: Dell R720XD, 2 Intel E5-2697 v2’s, 384G Ram, 27 TB Raid 5 w/Hot Spare.
  • Morgan: Dell R720XD, 2 Intel E5-2697 v2’s, 384G Ram, 27 TB RAID 5 w/Hot Spare.

UPS

  • Three APC Back-UPS [XS]1500 split between the four servers for uninterrupted power. Lasts about 10 minutes. Sufficient time to run the Ansible playbooks to shut down all the servers before the power goes out.

Software

I bought the VMware package from VMUG so I have license keys for a bunch of stuff. vCenter is limited to 6 CPUs. Still trying to figure out Distributed Switches. Each time I try, it fails and I have to rebuild the cluster.

All four servers are booting off an internal Sandisk Cruzer 16G USB thumb drive.

  • vSphere 7
  • vCenter 7

Most of what I’m doing fits into two categories. Personal stuff and a duplication of work stuff in order to improve my skills.

I have about 150 virtual machines as of the last time I checked plus another 100 for test builds on the KVM sandbox server. Most of my VMs are CentOS or Red Hat since I work in a Red Hat shop, and a few Ubuntu, a SUSE, one Solaris, one OpenBSD, and a couple of Windows workstations. I’m going to set up a Windows server for my DBA wife.

Personal

This consists of 12 servers.

  • pfSense. Firewall plus other internal stuff like DNS and Load Balancing. I have all servers cabled to the Internet and House Wifi so I can move pfSense to any of the three to restore access.
  • Jump Servers. I have three jump servers I use basically for utility type work. My Ansible playbooks are on these servers.
  • Hobby Software Development. This is kind of a dual purpose thing. I’m basically trying to duplicate how work builds projects by applying the same tools to my home development process. CI/CD: gitlab, jenkins, ansible tower, and artifactory. Development: original server, git server, and Laravel server for a couple of bigger programs
  • Identity Management server. Centralized user management. All servers are configured.
  • Spacewalk. I don’t want every server downloading packages from the ‘net. I’m on a high-speed wifi setup where I live. So I’m downloading the bulk of packages to the Spacewalk server and upgrading using it as the source. I have a Katello server that I want to get working due to the addition of Red Hat 8 servers.
  • Inventory. This is a local installation of the inventory program I wrote for my work servers. This has my local servers though. Basically it’s the ‘eat your own dog food’ sort of thing. 🙂
  • Media Servers. With almost 8 TB of space in use, I have two servers to split them between the R720XD’s. Movie Server. About 3 or so TB of movies I’ve ripped from my collection. Television Server. About 5 TB of television shows I’ve ripped from my collection.
  • Backups. I have two backup servers. One for local/desktop backups via Samba and one for remote backups of my physical server which is hosted in Florida.
  • Windows XP. I have two pieces of hardware that are perfectly fine but only work on XP so I have an XP workstation so I can keep using the hardware.
  • Windows 7. Just to see if I could really 🙂
  • Windows 98. I found the CD so I figured, why not?
  • Grafana. I’ve been graphing out portions of the environment but am still in the learning phase.

Work/Skills

There are four environments here. Development (38 servers), QA (18 servers), Staging (18 servers), and Production (52 servers).

  • Work development servers. The scripts and code I write at work, backed up to this server and also spread out amongst my servers.
  • Amazon AWS specific server. For testing building EC2 and EKS servers on AWS.
  • Nagios Servers. I have 6. One to monitor my Florida server, One to monitor my personal servers (above), and four to monitor my Work type servers. All six monitor the other five servers so if one bails, I’ll get notifications.
  • NMIS Servers. Work is using NMIS so I thought I’d spin one up to get familiar with the tool.
  • Docker Server. Basically learning docker.
  • Docker Registry. Holds my Docker images on prem so I’m not hitting the cloud every time I muck with a worker.
  • Kubernetes Servers. I have three Kubernetes clusters for various testing scenario. Three masters and three workers
  • OKD Servers. I have an OKD cluster, which is upstream OpenShift for testing an extended learning. Three masters and seven workers.
  • Elastic Stack clusters. This is a Kibana, multiple Logstash, and multiple Elastic Search servers. Basically centralized log management. Just like Kubernetes, three clusters for testing.
  • A Hashicorp Vault server for testing to see if it’ll work for what we need at work (secrets management).
  • Salt. One salt master for testing. All servers are connected.
  • Terraform. For testing VMware builds.
  • Jira server. Basically trying to get familiar with the software
  • Confluence. Again, trying to get used to it. I normally use a Wiki but work is transferring things over to Confluence.
  • Wiki. This is a duplicate of my work wikis, basically copying all the documentation I’ve written over the past few years.
  • Solaris 2540.

My wife is a DBA so I have a few database servers up, partly for her and partly for my use.

  • Cassandra
  • Postgresql.
  • MongoDB
  • Postgresql – This one I stood up for Jira and Confluence
  • Microsoft SQL Server
  • MySQL Master/Master cluster. This is mainly used by my applications but there for my wife to test things on as well.

The KVM server (Nikodemus) is more for Terraform testing. I’ve rebuilt my work server environment (just the OS part) and am currently rebuilding my various home environments to really dig into Infrastructure as Code. This is generally about 100 VMs depending on which environment I’m building.

Of course, this isn’t an exhaustive list of the VMs I’m running. I’m constantly spinning up VMs for testing purposes. Some stay up, some are shut down after I’m done mucking with it.

Desktop

And what would it be without seeing my workstation. In this picture, the center monitor is currently switched to my work laptop, the Mac on the desk. I just switch between monitor inputs to see my work or home system.

  • Computer Case: Thermaltake LEVEL 10 GT, White
  • Power Supply: EVGA Supernova 850 watt G2 80 Plus Gold
  • Motherboard: Asus Maximus VIII Hero
  • CPU: Intel 4 Core i7-6700K – socket FCLGA1151
  • CPU Cooler: Corsair H90 (Hydro 90; Water based cooler)
  • Memory: G.Skill Ripjaws V Series DDR4 PC2666 (4x8G: 32GB)
  • Video: GeForce RTX 2070 Super Gaming X 8G GDDR6
  • Video: GeForce RTX 2080 Ti Gaming X Trio 11G GDDR6
  • Storage: Samsung 850 EVO (500GB SSD) (Boot disk)
  • Storage: Samsung 850 EVO (500GB SSD) (Steam disk)
  • Storage: Samsung 870 EVO (1TB SSD) (Home disk)
  • Storage: Seagate 2TB Internal Desktop Hard Drive- 3.5″ Form Factor, SATA III 6 Gb/s, 64 MB Cache (x2) (Home and F:)
  • Storage: Seagate 3TB Internal Desktop Hard Drive- 3.5″ Form Factor, SATA III 6 Gb/s, 64 MB Cache (G:)
  • Monitor: Wasabi Mango UHD430 Real 4K HDMI 2.0 SE 43″ LG AH-IPS Panel UHD 3840×2160 Displayport 1.2 43-Inch 10Bit Monitor
  • Monitor: Acer G235H (4x)
  • Keyboard: Corsair K95 RGB Platinum XT
  • Mouse: Logitech G900
  • WebCam: Logitech WebCam Pro 9000
  • Speakers: Logitech X-540
  • OS: Microsoft Windows 10 Pro
  • Blu-Ray: LG Black Blu-ray Disc Drive SATA Model UH12NS30

I will note that I’m 64 and have been mucking about with computers for about 40 years now. I have several expired and active certifications. 3Com 3Wizard, Sun System and Network Admin, Cisco CCNA and CCNP, Red Hat RHCSA, and most recently a CKA, CKAD, and Amazon Certified Cloud Practitioner.

Posted in Computers | Tagged | Leave a comment

Installing OKD4

Posted on May 11, 2021 by Freejack

Overview

What I’m trying to accomplish here is to set up a homelab OpenShift type environment. I’m installing OKD4.7 in this case but OKD is an upstream of the Red Hat OpenShift so should be just fine for my use-case.

This installation is focused on my own requirements, as such I don’t have near the detail the Red Hat documentation has. This has been extremely helpful in understanding the Red Hat installation process.

I intend on documenting what I currently have, environment wise, and how to configure it appropriately. If we don’t have a similar environment, I highly recommend the article in the References section below. It greatly helped me understand the installation process.

Environment

  • I am running this project initially on VMware but will also be working on a second KVM server.
  • I currently run a pfSense package to manage my Firewall plus the DNS and DHCP services which are used for this project.
  • I’m using HAProxy to provide Load Balancing for the API-Server in the cluster. This is running on CentOS 8 Server.
  • I’m using a Service box to provide the necessary web server and NFS storage services. This is running on CentOS 8.
  • For the bootstrap node and cluster elements, I’ll be using a Fedora CoreOS image.

Virtual Machines

The servers are built in my Development environment (192.168.101.0/24). The assumption for this build is you already have the Gold Images for the non CoreOS nodes. Building the Operating Systems is beyond the scope of this article. Note that the function of the two non CoreOS nodes is pretty minor. The CPU, RAM, and Storage settings for the cluster are the recommendations from Red Hat.

Some general information to make the table a bit smaller.

  • Bootstrap/Control Plane – 4 CPU, 16 Gigs of RAM, 120 Gigs of storage.
  • Compute – 2 CPU, 8 Gigs of RAM, 120 Gigs of storage.
  • Operating System is Fedora CoreOS for the above three server types.
  • For the service and haproxy node, CentOS 8, 2 CPU, 4 Gigs RAM, and 30 Gigs of storage.
  • For the Machine Names, all are prefixed with bldr0cuomokd. So boot is bldr0cuomokdboot.
Machine TypeMachine NameIP AddressMAC
Bootstrap Serverboot192.168.101.10700:50:56:b3:a3:7e
Control Planemst1 192.168.101.101 00:50:56:b3:02:f4
Control Planemst2 192.168.101.102 00:50:56:b3:2a:62
Control Planemst3 192.168.101.103 00:50:56:b3:1a:11
Computewrk1 192.168.101.104 00:50:56:b3:42:30
Computewrk2 192.168.101.105 00:50:56:b3:8b:67
Computewrk3 192.168.101.106 00:50:56:b3:d8:81
Computewrk4 192.168.101.109 00:50:56:91:93.c3
Computewrk5 192.168.101.110 00:50:56:91:a7:21
Computewrk6 192.168.101.111 00:50:56:91:2f:be
Computewrk7 192.168.101.112 00:50:56:91:7e:7a
NFS/Websvc1 192.168.101.100 00:50:56:b3:38:31
HAProxyhap1 192.168.101.108 00:50:56:b3:e0:0a

Note that worker nodes 4 and 5 were added after the cluster was built but prior to the 24 hour period expiring. Worker nodes 6 and 7 were added after the 24 hour window expired.

DNS

The above Machine Names have been entered into DNS. In addition, the following aliases need to be added to DNS.

The cluster domain name will be [site].internal.pri. For example, bldr0-0.internal.pri.

  • api – The api server. Aliased to the haproxy hostname.
  • api-int – The internal api server alias. Aliased to the haproxy hostname.
  • console-openshift-console – Console access to the cluster. Aliased to the haproxy hostname.
  • oauth-openshift – The authentication method. Aliased to the haproxy hostname.
  • etcd-0 – The first etcd node. Aliased to the first master node.
  • etcd-1 – The second etcd node. Aliased to the second master node.
  • etcd-2 – The third etcd node. Aliased to the third master node.
  • bootstrap – The bootstraper server. Aliased to the bootstrap server.
  • address=/okd.internal.pri/192.168.101.108 – This needs to be added to the Options in the DNS Forwarder section of pfSense. This lets the ingress router work.

Note that I used okd.internal.pri as I didn’t quite understand the naming convention for the domain. If I were to create clusters in all four of my environments, I’d be using bldr0-0, cabo0-0, tato0-0, and lnmt1-2 which matches my existing Kubernetes cluster naming convention.

DHCP

The components will start and will need to get an IP address from the DHCP server. The IP will need to be external to the DHCP server range and will require the MAC address in order for discovery to work.

Note that I used the wrong MAC for the last two worker nodes and the IPs came up in the DHCP range. So ensure you have the correct MAC address (apparently VMware uses 00:50:56:b3:00:00 and 00:50:56:91:00:00, note the b3 and 91 differences). Just a note of caution.

Fedora CoreOS

We’ll be downloading two images. You’ll need the ISO image so you can boot to it to start the installation and you’ll need the raw image for the actual component installation.

On Fedora CoreOS Download Site click on the Bare Metal &Virtualized tab and then down load the ISO image plus the signature and sha256 keys. In addition, download the raw image plus the signature and sha256 keys.

Prepare Nodes

Again, I’m on a VMware environment. To prepare for the cluster build, create 7 blank VMs using the configuration as noted in the Virtual Machines section. In the VM Options tab under Boot Options, check the Force BIOS Setup checkbox. This lets you attach the CoreOS image to the server so you can install CoreOS.

Load Balancer

You need to install haproxy on the HAProxy node and configure it to be a Load Balancer for the cluster. 

# yum install -y haproxy

For the global and defaults sections, increase the maxconn line to 20000.

Delete everything after the end of the defaults section.

For statistics, add: 

listen stats
    bind :9000
    mode http
    stats enable
    stats uri /

For the API Server (port 6443), add the following front end and back end sections: 

frontend okd4_k8s_api_fe
    bind :6443
    default_backend okd4_k8s_api_be
    mode tcp
    option tcplog

backend okd4_k8s_api_be
    balance source
    mode tcp
    server      bldr0cuomokdboot 192.168.101.107:6443 check
    server      bldr0cuomokdmst1 192.168.101.101:6443 check
    server      bldr0cuomokdmst2 192.168.101.102:6443 check
    server      bldr0cuomokdmst3 192.168.101.103:6443 check

When the servers are being built, the following section will be needed: 

frontend okd4_machine_config_server_fe
    bind :22623
    default_backend okd4_machine_config_server_be
    mode tcp
    option tcplog

backend okd4_machine_config_server_be
    balance source
    mode tcp
    server      bldr0cuomokdboot 192.168.101.107:22623 check
    server      bldr0cuomokdmst1 192.168.101.101:22623 check
    server      bldr0cuomokdmst2 192.168.101.102:22623 check
    server      bldr0cuomokdmst3 192.168.101.103:22623 check

For port 80 traffic, if any, the following section is needed: 

frontend okd4_http_ingress_traffic_fe
    bind :80
    default_backend okd4_http_ingress_traffic_be
    mode tcp
    option tcplog

backend okd4_http_ingress_traffic_be
    balance source
    mode tcp
    server      bldr0cuomokdwrk1 192.168.101.104:80 check
    server      bldr0cuomokdwrk2 192.168.101.105:80 check
    server      bldr0cuomokdwrk3 192.168.101.106:80 check

And for port 443 traffic, the following section is needed: 

frontend okd4_https_ingress_traffic_fe
    bind *:443
    default_backend okd4_https_ingress_traffic_be
    mode tcp
    option tcplog

backend okd4_https_ingress_traffic_be
    balance source
    mode tcp
    server      bldr0cuomokdwrk1 192.168.101.104:443 check
    server      bldr0cuomokdwrk2 192.168.101.105:443 check
    server      bldr0cuomokdwrk3 192.168.101.106:443 check

Once edited, enable and start haproxy. 

systemctl enable haproxy
systemctl start haproxy

Web Server

In order for the OpenShift nodes to retrieve the image and ignition files, you’ll need to install a web server on the Service node. Also configure the node to listen on port 8080. 

dnf install -y httpd
sed -i 's/Listen 80/Listen 8080/' /etc/httpd/conf/httpd.conf

Once done, enable and start the server. 

systemctl enable httpd
systemctl start httpd

OpenShift Binaries

You need to have the oc binary plus the openshift-installer on your Service Node.

OpenShift Downloads

While not necessary for the installation, you should retrieve the kubectl binary as well.

OpenShift install-config.yaml File

The following file is used for building the cluster. 

apiVersion: v1
baseDomain: [domain]                     # update the domain info
metadata:
  name: [sub-domain]                     # update the sub-domain info

compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0

controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3

networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14                   # Verify your network
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16                         # Verify the service network.

platform:
  none: {}

fips: false

pullSecret: '{"auths": ...}'               # add the pullSecret
sshKey: 'ssh-ed25519 AAAA...'              # add your ssh public key

OpenShift Pull Secret

In the configuration file, update the pullSecret from the download below.

OpenShift Pull Secret Download

SSH Key

When building the compute nodes, in order to access the servers, you’ll need to provide an SSH Key from the account that will need to access the servers. If you don’t already have a public key ready, generate one. 

ssh-keygen -t rsa

To log in to the servers, log in as core@[servername].

Perform The Installation

Create an install_dir directory and make a copy of the install-config.yaml file into this directory. Then run the installer. 

openshift-install create manifests --dir=install_dir
INFO Consuming Install Config from target directory
INFO Manifests created in: install_dir/manifests and install_dir/openshift

Rename the install_dir to the manifests directory.

Run Installer Again

This time you’re creating the ignition-configs. Note that the ignition files contain a certificate that expires in 24 hours. If you need to rebuild the cluster or add nodes after the 24 hours has expired, you’ll need a new certificate. See the Adding Workers section at the end of this document.

$ openshift-install create ignition-configs --dir=install_dir/
INFO Consuming Install Config from target directory
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
INFO Ignition-Configs created in: install_dir and install_dir/auth

Install Configuration

In the /var/www/html directory, create the okd4 directory, copy the new configuration files, copy the fedora CoreOS files and rename them (makes it easier to type on the console), and set the permissions. 

mkdir /var/www/html/okd4
cp -r manifests/* /var/www/html/okd4/
cp -r install_dir/* /var/www/html/okd4/
mv fedora-coreos-33.20210426.3.0-metal.x86_64.raw.xz /var/www/html/okd4/fcos.raw.xz
mv fedora-coreos-33.20210426.3.0-metal.x86_64.raw.xz.sig /var/www/html/okd4/fcos.raw.xz.sig 
chown -R apache: /var/www/html
chmod -R 755 /var/www/html

Build The bootstrap Server

You’ll need to boot into the Fedora CoreOS Live image, press tab to jump to the kernel line, and enter in the following information: 

coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.101.100:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.101.100:8080/okd4/bootstrap.ign

Now The Control Plane

Start the master servers and after hitting tab, enter the following lines. 

coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.101.100:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.101.100:8080/okd4/master.ign

It can take 10 or 15 minutes for the masters to register.

Now The Compute Nodes

Same here, start the workers and enter the following lines

coreos.inst.install_dev=/dev/sda
coreos.inst.image_url=http://192.168.101.100:8080/okd4/fcos.raw.xz
coreos.inst.ignition_url=http://192.168.101/100:8080/okd4/worker.ign

It can take quite some time for the workers to register, up to 30 minutes.

Update HAProxy

When all nodes have been bootstrapped, you need to remove the bootstrap entry in the HAProxy configuration. Just comment it out and restart haproxy.

Certificate Signing Requests

Once the workers have been accepted into the cluster, you’ll need to approve any CSRs so they can start loading up pods. You’ll need to get the jq tool so you can approve a bunch of CSRs in a bundle. 

wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
chmod +x jq
sudo mv jq /usr/local/bin/
jq --version

Once you have the jq file, check for pending CSRs. At the start, there will be a ton. For new nodes, it can take several minutes as the new nodes tend to upgrade the CoreOS OS during the bootstrap process. Once things settle out, new CSRs should be pending.

oc get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                                                                   CONDITION
csr-6n8c6   91s     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-85mmn   31m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-8tn26   6m38s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
csr-lgxlv   16m     kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

Once you see the pending CSRs, then run this command to approve all pending CSRs 

oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve
certificatesigningrequest.certificates.k8s.io/csr-6n8c6 approved
certificatesigningrequest.certificates.k8s.io/csr-85mmn approved
certificatesigningrequest.certificates.k8s.io/csr-8tn26 approved
certificatesigningrequest.certificates.k8s.io/csr-lgxlv approved

For some things such as new worker nodes, it will take two passes to approve all the CSRs.

Console Access

Finally check the status of the clusteroperators, specifically the console. Once it’s up and running, you can get your password from the install_dir/auth/kubeadmin_password file. Log in to the console as kubeadmin and you’re in! 

$ oc get clusteroperators
NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.okd-2021-04-24-103438   True        False         False      13h
baremetal                                  4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
cloud-credential                           4.7.0-0.okd-2021-04-24-103438   True        False         False      5d16h
cluster-autoscaler                         4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
config-operator                            4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
console                                    4.7.0-0.okd-2021-04-24-103438   True        False         False      4d23h
csi-snapshot-controller                    4.7.0-0.okd-2021-04-24-103438   True        False         False      4d23h
dns                                        4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
etcd                                       4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
image-registry                             4.7.0-0.okd-2021-04-24-103438   True        False         True       2d17h
ingress                                    4.7.0-0.okd-2021-04-24-103438   True        False         True       5d14h
insights                                   4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
kube-apiserver                             4.7.0-0.okd-2021-04-24-103438   True        False         False      5d13h
kube-controller-manager                    4.7.0-0.okd-2021-04-24-103438   True        False         False      5d13h
kube-scheduler                             4.7.0-0.okd-2021-04-24-103438   True        False         False      5d14h
kube-storage-version-migrator              4.7.0-0.okd-2021-04-24-103438   True        False         False      13h
machine-api                                4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
machine-approver                           4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
machine-config                             4.7.0-0.okd-2021-04-24-103438   True        False         False      46h
marketplace                                4.7.0-0.okd-2021-04-24-103438   True        False         False      4d23h
monitoring                                 4.7.0-0.okd-2021-04-24-103438   True        False         False      45h
network                                    4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
node-tuning                                4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
openshift-apiserver                        4.7.0-0.okd-2021-04-24-103438   True        False         False      13h
openshift-controller-manager               4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
openshift-samples                          4.7.0-0.okd-2021-04-24-103438   True        False         False      5d13h
operator-lifecycle-manager                 4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
operator-lifecycle-manager-catalog         4.7.0-0.okd-2021-04-24-103438   True        False         False      5d15h
operator-lifecycle-manager-packageserver   4.7.0-0.okd-2021-04-24-103438   True        False         False      13h

Persistent Storage

Next up is to create some persistent storage.

NFS Server

In order to set up a persistent volume, you’ll need to configure an NFS mount on an accessible server.

On the Service server: 

dnf install -y nfs-utils
systemctl enable nfs-server rpcbind
systemctl start nfs-server rpcbind
mkdir -p /var/nfsshare/registry
chmod -R 777 /var/nfsshare
chown -R nobody:nobody /var/nfsshare

Then set up the share. 

echo '/var/nfsshare 192.168.101.0/24(rw,sync,no_root_squash,no_all_squash,no_wdelay)' > /etc/exports

Assuming selinux and a firewall is running, you’ll need to make the following changes. Ignore the setsebool and firewall-cmd lines if one or both are not configured. 

sudo setsebool -P nfs_export_all_rw 1
sudo systemctl restart nfs-server
sudo firewall-cmd --permanent --zone=public --add-service mountd
sudo firewall-cmd --permanent --zone=public --add-service rpc-bind
sudo firewall-cmd --permanent --zone=public --add-service nfs
sudo firewall-cmd --reload

Image Registry

Apply the following registry file to the cluster. Make sure the server IP is accurate. 

$ cat registry_pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /var/nfsshare/registry
    server: 192.168.101.100

And

$ oc apply -f registry_pv.yaml

By default, persistent storage isn’t configured. As such the operator sets it up as ‘Removed’. You’ll need to edit the configuration to tell OpenShift that persistent storage is available.

$ oc edit configs.imageregistry.operator.openshift.io

Update the following settings. 

managementState: Removed

  storage: {}

Change to: 

managementState: Managed

  storage:
    pvc:
      claim:

Create Accounts

In order to create accounts for users to access the cluster, you use the htpasswd program and submit it to the cluster. You’ll also need to create a rolebinding or clusterrolebinding to provide permissions.

htpasswd

Simply create a file that contains your username and password for accessing the OKD cluster.

  • -c = Create New File. This does overwrite an existing file so use caution.
  • -B = use most secure algorithm.
  • -b = Accept the username and userpassword on the command line.

Create Secret

Next create the htpass-secret secret. 

oc create secret generic htpass-secret --from-file=htpasswd=htpasswd -n openshift-config

Provider

You’ll need to apply a provider for the htpasswd supplied credentials. Use the following file. This says the OAuth object will use the htpass-secret for credentials. 

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
  name: cluster
spec:
  identityProviders:
  - name: htpasswd_provider
    mappingMethod: claim
    type: HTPasswd
    htpasswd:
      fileData:
        name: htpass-secret

Bindings

If you log in as yourself to the console now, you can access but have no authority to see or do anything. You’ll need to bind your account to a RBAC set of permissions. 

oc adm policy add-cluster-role-to-user cluster-admin [username]

And now I have access.

Adding Workers

When the ignition files were initially created, they have a certificate that expires 24 hours after it’s been created. Within that window, you can add multiple worker nodes. After the certificate expires, you’ll need to get a new certificate from the cluster. This certificate expires in 10 years and you can add multiple worker nodes from then on.

Things to remember:

  • Add the bootstrap servers back into the haproxy server configuration and restart.
  • Also add the new workers to the worker section in the haproxy configuration.
  • Create the DNS entries.
  • Update the DHCP configuration for the new Workers.

Within The Window

You can simply add the new Worker Nodes by following the above instructions for creating a blank VM, starting it with the Fedora CoreOS Live ISO, and entering in the kernel parameters to bootstrap the Workers. The worker.ign files are valid for any number of Worker Nodes.

Window Has Expired

You need to extract the new certificate from the cluster and add it to the worker.ign file. Make sure you back up the current worker.ign file, just in case.

Extract certificate: 

openssl s_client -connect api-int.okd.internal.pri:22623 -showcerts </dev/null 2>/dev/null|openssl x509 -outform PEM > api-int.pem

The creates the api-int.pem file. This file now needs to be converted into a base64 string. The –wrap=0 parameter turns the block into a single line. 

base64 --wrap=0 ./api-int.pem 1> ./api.int.base64

Now back up the worker.ign file. 

cp worker.ign worker.ign.backup

And replace the current certificate with the new one located in the api.int.base64 file. 

{"ignition":{"config":{"merge":[{"source":"https://api-int.okd.internal.pri:22623/config/worker"}]},"security":{"tls":{"certificateAuthorities":[{"source":"data:text/plain;charset=utf-8;base64,[ADD CERTIFICATE HERE"}]}},"version":"3.2.0"}}

And finally follow the process above to add a new worker to the cluster.

References

I used the following link as it was more focused on what I’m running vs the Red Hat site which has a ton of options to consider. I’ve built the cluster three times now and with the third build, I rewrote this article as instruction for my specific use-case. If you have a similar environment, this article might be helpful. The following link lets you create a similar environment but firewalled away from your central homelab environment. Ultimately it made me a bit more skilled and able to better understand the more extensive Red Hat docs.

The following links are the Red Hat docs for a Bare Metal deployment and copies of the links in the above article so you’re not hunting for the pullSecret link.

Posted in Computers, OpenShift | Tagged , , , | 1 Comment

LACP, Bonding, Bridges, and VLAN Tagging

Posted on April 18, 2021 by Freejack

Overview

This article provides some brief information on setting up bonded interfaces in Linux, configuring Link Aggregation Control Protocol (LACP), and then setting up tagged VLANs. Many times there’s an assumption of knowledge. This article provides some background and resources to verify the knowledge or increase it if some information is lacking. This is not intended, however, to be a detailed or deep dive into networking in Linux. Just to provide some information and configurations for the current requirement. There are other settings and options available. Feel free to continue reading on the topics and even take a class for even more information.

As always links to some of the pages I used for this article are in the references section at the end. If you find a better link, by all means, let me know and I’ll add it to the references.

Link Aggregation Control Protocol

We’ll start with LACP. We use LACP on a Bonded Interface in order to provide redundancy, load balancing, and to improve performance. A collection of LACP interfaces is called a Link Aggregation Group (LAG). Unlike other options, LACP collects all interfaces into a single, wider trunk for traffic. If we have 4 gigabit interfaces, we have a 4 gigabit trunk and not 4 individual 1 gigabit pipes. In my example below, I have 12 network interfaces as a LAG for bond0 so a 12 gigabit trunk.

mode

To enable LACP, set the mode to 802.3ad.

lacp_rate

This flag has two options, 0 and 1. If set to 0 (slow; default setting), the default LACP packet is sent to a peer (like the switch) every 30 seconds. If set to 1 (fast), the default LACP packet is sent to a peer every 1 second.

xmit_transfer_policy

There are three policy settings for this option.

  • layer2 (Data Link) – The default policy. Transmission hash is based on MAC. All network traffic is on a single slave interface in the bonded LACP.
  • layer2+3 (Data Link + Network) – Transmission hash is based on MAC and IP address. Sends traffic to destinations on the same slave interface. More balanced so provides the best performance and stability.
  • layer3+4 (Network + Transport) – Creates a transmission hash based on the upper network layer (Transport) whenever possible. It allows multiple traffic or connections spanning over multiple slave interfaces in a bonded LACP. One connection will not span over multiple interfaces. Reverts to layer2 (Data Link) for non IP traffic. Note that this isn’t fully compliant with LACP. This can be the best option for performance and stability but as noted, it’s not fully LACP compliant.

miimon

This is how often the bond checks the bonded interfaces to ensure they’re still active. A good value is 100 (milliseconds) which is the default if not set.

You can verify the setting by checking.

$ cat /sys/devices/virtual/net/bond0/bonding/miimon 100

Bonded Interfaces

Network Bonding is essentially combining multiple network interfaces into a single interface. There are several modes you can select to manage the behavior of the bonded interface. For our purposes, we’re using mode 4 which is the 802.3ad specification, Dynamic Link Aggregation.

Bonding Module

It’s very likely that whatever system you’re going to set up bonding on will be configured. But just to be sure, you’ll want to verify it’s available.

# modinfo bonding
filename:       /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/drivers/net/bonding/bonding.ko.xz
author:         Thomas Davis, tadavis@lbl.gov and many others
description:    Ethernet Channel Bonding Driver, v3.7.1
version:        3.7.1
license:        GPL
alias:          rtnl-link-bond
retpoline:      Y
rhelversion:    7.4
srcversion:     33C47E3D00DF16A17A5AB9C
depends:
intree:         Y
vermagic:       3.10.0-693.21.1.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        03:DA:60:92:F6:71:13:21:B5:AC:E1:2E:84:5D:A9:73:36:F7:67:4D
sig_hashalgo:   sha256
...

 If modinfo doesn’t return any information, you’ll want to activate bonding.

# modprobe --first-time bonding

Network Interfaces

The servers have a 4 port network card in addition to the four ports on the motherboard. I will be creating a 8 network interface bond.

  • eno5-eno8 – On Board
  • ens1f0-ens1f3 – Network Card

Configurations

For CentOS, multiple files are created in the /etc/sysconfig/network-scripts directory for the bond0 interface and each interface as listed above.

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BONDING_MASTER=yes
IPADDR=192.168.1.10
PREFIX=24
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3"

For CentOS, we’ll create a file for each interface. This is just the first file but each interface file will be the same except for the DEVICE and NAME entries.

# vi /etc/sysconfig/network-scripts/ifcfg-eno1
TYPE="Ethernet"
BOOTPROTO="None"
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV4INIT="no"
DEVICE="eno1"
NAME="eno1"
ONBOOT="yes"
MASTER="bond0"
SLAVE="yes"

For Ubuntu which I’m using as my KVM server, you’ll edit the single /etc/netplan/01-netcfg.yaml file. Remember that spacing is critical when working in yaml. Initially you’ll disable dhcp4 for all interfaces. Then set up the bonding configuration.

$ cat 01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: no
    eno2:
      dhcp4: no
    eno3:
      dhcp4: no
    eno4:
      dhcp4: no
    enp8s0f0:
      dhcp4: no
    enp8s0f1:
      dhcp4: no
    enp9s0f0:
      dhcp4: no
    enp9s0f1:
      dhcp4: no
    enp12s0f0:
      dhcp4: no
    enp12s0f1:
      dhcp4: no
    enp13s0f0:
      dhcp4: no
    enp13s0f1:
      dhcp4: no

  bonds:
    bond0:
      interfaces:
      - eno1
      - eno2
      - enp8s0f0
      - enp8s0f1
      - enp12s0f0
      - enp12s0f1
      parameters:
        mode: 802.3ad
        transmit-hash-policy: layer3+4
        mii-monitor-interval: 100

Bond Status

You can check the bonding status by viewing the current configuration in /proc.

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: be:70:a3:ab:10:6d
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 1
        Actor Key: 9
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: enp13s0f1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:ca:12:45
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: churned
Actor Churned Count: 0
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: be:70:a3:ab:10:6d
    port key: 9
    port priority: 255
    port number: 1
    port state: 77
details partner lacp pdu:
    system priority: 65535
    system mac address: 00:00:00:00:00:00
    oper key: 1
    port priority: 255
    port number: 1
    port state: 1

...

Bridge Interfaces

A Bridge interface connects two networks together. Generally you’re connecting a VM network such as with KVM with the physical or bonded interface.

TYPE=Bridge
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=br700
DEVICE=br700
ONBOOT=yes
IPADDR=192.168.1.10
PREFIX=24
GATEWAY=192.168.1.1
NM_CONTROLLED=no
USERCTL=no
DNS1=192.168.1.254
DOMAIN=internal.pri

You can get some bridge information using the brctl command.

# brctl show
bridge name	bridge id		STP enabled	interfaces
br2719		8000.98f2b3288d26	no		bond2.719
							vnet1
br2751		8000.98f2b3288d26	no		bond2.751
							vnet5
br2752		8000.98f2b3288d26	no		bond2.752
							vnet6
br700		8000.98f2b3288d24	no		bond0.700
							vnet0
							vnet11
							vnet13
							vnet2
							vnet20
							vnet21
							vnet3
							vnet7
							vnet8
br710		8000.98f2b3288d25	no		bond1.710
							vnet10
							vnet12
							vnet19
							vnet22
							vnet4
							vnet9
br750		8000.98f2b3288d24	no		bond0.750
virbr0		8000.525400085dad	yes		virbr0-nic
virbr1		8000.525400a5a4ea	yes		virbr1-nic

When using nmcli to create a bridge, you can connect an interface that doesn’t have a connection profile or connect an interface with an existing connection profile. A connection profile is an interface that was created by Network Manager.

First create the bridge interface.

# nmcli con add type bridge con-name br810 ifname br810

For an interface without a connection profile.

# nmcli con add type ethernet slave-type bridge con-name br810-eno5 \
    ifname eno5 master br810
# nmcli con add type ethernet slave-type bridge con-name br810-eno6 \
    ifname eno6 master br810

However if we already have a connection profile, you just designate the bridge as the master.

# nmcli con mod bond0.810 master br810

For Ubuntu systems, you’ll again edit the /etc/netplan/01-netcfg.yaml file to add the bridge information.

bridges:
    br0:
      dhcp4: false
      addresses:
        - 192.168.1.10/24
      gateway4: 192.168.1.254
      nameservers:
        addresses:
          - 192.168.1.254
        search: ['internal.pri schelin.org']
      interfaces:
        - bond0

VLAN Tagging

VLAN Tagging simply lets you tag traffic associated with the underlying interface so that the switch knows where traffic needs to be sent.

Configuration

This will describe setting up one of the Dell servers. This is the interface hierarchy. At the top level is the bridged interface followed by the bonded and then physical interfaces.

  • br810 – Bridged interface. This has the IP Assignments.
    • bond0.810 – Bonded VLAN tagged interface, references br810.
      • bond0 – The underlying bonded interface. This has the BONDED_OPTS line.
        • eno5 – The lower physical interface and references bond0.
        • eno6 – The lower physical interface.
        • eno7 – The lower physical interface.
        • etc…

Here is the list of IP addresses. This is for a dual interface system which my Dell is. For the Ubuntu based Dell box, it’s a single interface as described above. The only real change when viewing the examples below should be the IP address for each system.

Domain is internal.pri.

Current Server NameNew Server NameApp/IP AddressGatewayManagement/IP AddressGateway
morganniki192.168.1.10/24192.168.1.110.100.78.10010.100.78.254

The production Dell system has 8 interfaces. eno5-eno8 and ens1f0-ens1f3. We’ll set up the bridge, the bonded vlan tagged interface, the bonded interface, and then connect all the interfaces to the bond.

Since this is via the console, the example output below will not have all the information.

The configuration process is to create the bonded interface first. Then add the underlying physical interfaces. Next add a bridged interface. And finally configure the VLAN for the tagged interface.

# nmcli con add type bond con-name bond0 ifname bond0 \
    bond.options "mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer2+3" \
    ipv4.method disabled ipv6.method ignore

The bond0 interface doesn’t get an IP address, the bridge interface does. Hence the ipv4.method disabled option.

Next add the physical interfaces to the bond. Since they already have connection profiles, we don’t need to further configure the interfaces.

# nmcli con add type bond-slave ifname eno5 master bond0
# nmcli con add type bond-slave ifname eno6 master bond0
# nmcli con add type bond-slave ifname eno7 master bond0
# nmcli con add type bond-slave ifname eno8 master bond0
# nmcli con add type bond-slave ifname ens1f0 master bond0
# nmcli con add type bond-slave ifname ens1f1 master bond0
# nmcli con add type bond-slave ifname ens1f2 master bond0
# nmcli con add type bond-slave ifname ens1f3 master bond0

Now we configure a bridge interface. Make sure you change the IP address.

# nmcli con add type bridge con-name br810 ifname br810 ip4 192.168.1.10/24

We’ll need to add the gateway as well. And change the gateway address too.

# nmcli con mod br810 ipv4.method manual ipv4.gateway 192.168.1.1

Finally add the VLAN interface on top of the bond0 interface but with the bridge interface as the master.

# nmcli con add type vlan con-name bond0.810 ifname bond0.810 dev bond0 \
    vlan.id 810 master br810 slave-type bridge

That should bring the interface up. You can review the status by:

# more /proc/net/bonding/bond0

As the network is a service network, it’s not reachable so a second bridge and vlan has been configured. Make sure you change the IP address.

# nmcli con add type bridge con-name br700 ifname br700 ip4 10.100.78.100/24

Add the gateway. And change the gateway address too.

# nmcli con mod br700 ipv4.method manual ipv4.gateway 10.100.78.254

And the VLAN.

# nmcli con add type vlan con-name bond0.700 ifname bond0.700 dev bond0 \
    vlan.id 700 master br700 slave-type bridge

Finally a route is needed to ensure we can access the servers. Make sure you change that last IP to the gateway for the Management IP address.

# nmcli con mod br700 +ipv4.routes "192.168.0.0/16 192.168.1.1"

And remove the default route for br700. This changes the ‘DEFROUTE=yes’ option to ‘DEFROUTE=no’ and removes the gateway if set (as in above).

# nmcli con mod br700 ipv4.never-default yes

References

Posted in Computers, KVM | Leave a comment