Overview
There are multiple IP assignments used in Kubernetes. In addition to the internal networking (by Calico in this case), you can install an Ingress Controller to manage access to your applications. This article provides some basic Service information as I explore the networking and work towards exposing my application(s) externally using an Ingress Router.
Networking
You can manage traffic for your network by either using a Layer 2 or Layer 3 device or by using an Overlay network. Due to the requirement of maintaining pod networks in a switch, the easiest method is using an Overlay network. This encapsulates network traffic using VXLAN (Virtual Extensible LAN) and tunnels it to other worker nodes in the cluster.
Services
I’ll be providing information on the three ways to provide access to applications using a Service context and any plusses and minuses.
ClusterIP
When you create a service, the default configuration assigns a ClusterIP from a pool of IPs defined as the Service Network when you created the Kubernetes cluster. The Service Network is how pods communicate with each other in the cluster. In my configuration, 10.69.0.0/16 is the network I assigned to the Service Network. When I look at a set of services, every one will have a 10.69.0.0/16 IP address.
$ kubectl get svc
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.69.0.1 <none> 443/TCP 8d
default my-nginx NodePort 10.69.210.167 <none> 8080:31201/TCP,443:31713/TCP 11m
NodePort
Configuring a service with the type of NodePort is probably the easiest. You’re defining an externally accessible port in the range of 30,000-32,767 that is associated with your application’s port.
apiVersion: v1
kind: Service
metadata:
name: my-nginx
namespace: default
labels:
run: my-nginx
spec:
type: NodePort
ports:
- name: http
nodePort: 30100
port: 8080
targetPort: 80
protocol: TCP
- name: https
nodePort: 30110
port: 443
protocol: TCP
selector:
run: my-nginx
When you check the services, you’ll see the ports that were randomly assigned if you didn’t define them.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-nginx NodePort 10.69.91.108 <none> 8080:30100/TCP,443:30110/TCP 11h
Anyway, when using NodePort, you simply access the API Server IP Address and tack on the port. With that you have access to the application.
https://bldr0cuomvip1.dev.internal.pri:30110
The positive aspect here is regardless of which worker node the container is running on, you always have access. But the problem with this method is your load balancer has to know about the ports and update the configuration, plus your firewall has to allow access to either a range of ports or an entry for each port. Not killer but can complicate things especially if you’re not assigning the NodePort. Infrastructure as Code does help with managing the Load Balancer and Firewall configurations pretty well.
Side note, you can also access any worker node with the defined port number and Kubernetes will route you to the correct node. Certainly accessing the API server with the port number is optimum.
ExternalIPs
The use of externalIPs lets you access an application/container via the IP of the worker node the app is running on. You can set a new DNS entry so you can access the application without using a default port or the defined port (common ones being 8080 for example).
You’d update the above service to add the externalIPs line. This would be the IP of the worker node the container is running on. In order to add the line you’ll need to get the list of workers to see which node the container is running on.
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
curl 1/1 Running 1 (17h ago) 18h 10.42.251.135 bldr0cuomknode1.dev.internal.pri <none> <none>
curl-deployment-7d9ff6d9d4-jz6gj 1/1 Running 0 12h 10.42.251.137 bldr0cuomknode1.dev.internal.pri <none> <none>
echoserver-6f54957b4d-94qm4 1/1 Running 0 45h 10.42.80.7 bldr0cuomknode3.dev.internal.pri <none> <none>
my-nginx-66689dbf87-9x6kt 1/1 Running 0 12h 10.42.80.12 bldr0cuomknode3.dev.internal.pri <none> <none>
We see the my-nginx pod is running on bldr0cuomknode3.dev.internal.pri. Get the IP for it and update the service (I know all my K8S nodes are 160-162 for control and 163-165 for workers so knode3 is 165).
$ kubectl edit svc my-nginx
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2023-04-06T01:49:28Z"
labels:
run: my-nginx
name: my-nginx
namespace: default
resourceVersion: "1735857"
uid: 439abcae-94d8-4810-aa44-2992d7a30a63
spec:
clusterIP: 10.69.91.108
clusterIPs:
- 10.69.91.108
externalIPs:
- 192.168.101.165
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 32107
port: 8080
protocol: TCP
targetPort: 80
- name: https
nodePort: 31943
port: 443
protocol: TCP
targetPort: 443
selector:
run: my-nginx
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
Then add the externalIPs: line as noted above. When done, check the services
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver NodePort 10.69.249.118 <none> 8080:32356/TCP 45h
kubernetes ClusterIP 10.69.0.1 <none> 443/TCP 8d
my-nginx NodePort 10.69.91.108 192.168.101.165 8080:32107/TCP,443:31943/TCP 13h
If you check the pod output above, note that the echoserver is also on knode3 plus using port 8080. The issue here is containers can’t have services using the same ports as the first service will be the only one that responds. Either move the pod or change the port to a unique one.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver NodePort 10.69.249.118 192.168.101.165 8080:32356/TCP 45h
kubernetes ClusterIP 10.69.0.1 <none> 443/TCP 8d
my-nginx NodePort 10.69.91.108 192.168.101.165 8080:32107/TCP,443:31943/TCP 12h
Finally, the problem should be clear. If knode3 goes away, or goes into maintenance mode, or heck is replaced, the IP address is now different. You’ll need to check the pods, update the service to point to the new node, then update DNS to use the new IP address. And depending on the DNS TTL, it could take some time before the new IP address is returned. Also what if you have more than one pod for load balancing or if you’re using Horizontal Pod Autoscaling (HPA)?
Ingress-Controllers
I checked out several ingress controllers and because Openshift is using a HAProxy ingress controller, that’s what I went with. There are several others of course and you’re free to pick the one that suits you.
The benefit of an Ingress Controller is it combines the positive features of a NodePort and an ExternalIP. Remember with a NodePort, you access your application by using the Load Balancer IP or Worker Node IP, but with a unique port number. It’s annoying because you have to manage firewalls for all the ports. With an ExternalIP, you can assign that to a Service and create a DNS entry to point to that IP and folks can access the site through a well crafted DNS entry. The problem of course if if the node goes away, you have to update the DNS with the new node IP where the pod now resides.
An Ingress Controller installs the selected ingress pod which has a label. Then you create an Ingress route using that label in a metadata.annotation, then create a DNS entry that points to the Load Balancer IP. The Ingress route knows about the DNS entry and has the label so points the incoming traffic to the Ingress Controller which then sends traffic to the appropriate pod or pods regardless of worker.
Ingress Controller Installation
I’ve been in positions where I can’t use helm so I’ve not used it much but the haproxy-ingress controller is only installable via helm chart so this is a first for me. First add the helm binary, then the helm charts for the controller.
helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts
Next is to create a custom values file, I called it haproxy-ingress-values.yaml
controller:
hostNetwork: true
Then install the controller. This creates the ingress-controller namespace.
helm install haproxy-ingress haproxy-ingress/haproxy-ingress\
--create-namespace --namespace ingress-controller\
--version 0.14.2\
-f haproxy-ingress-values.yaml
And that’s all there is to it. Next up is creating the necessary ingress rules for applications.
Ingress Controller Configuration
I’m going to be creating a real basic Ingress entry here to see how things work. I don’t need a lot of options but you should check out the documentation and feel free to adjust as necessary for your situation.
Initially I’ll be using a couple of examples I used when testing this process. In addition I have another document I used when I was managing Openshift which gave me the little hint on what I was doing wrong to this point.
There are two example sites I’m using to test this. One is from the kubernetes site (my-nginx) and one is from the haproxy-ingress site (echoserver) both linked in the References section.
my-nginx Project
The my-nginx project has several configuration files that make up the project. The one thing it doesn’t have is the ingress.yaml file needed for external access to the site. Following are the configurations used to build this site.
The configmap.yaml file provides data for the nginx web server.
apiVersion: v1
kind: ConfigMap
metadata:
name: nginxconfigmap
data:
default.conf: |
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
listen 443 ssl;
root /usr/share/nginx/html;
index index.html;
server_name localhost;
ssl_certificate /etc/nginx/ssl/tls.crt;
ssl_certificate_key /etc/nginx/ssl/tls.key;
location / {
try_files $uri $uri/ =404;
}
}
For the nginxsecret.yaml Secret, you’ll first need to create a couple of ssl certificates using the openssl command.
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout /var/tmp/nginx.key -out /var/tmp/nginx.crt \
-subj "/CN=my-nginx/O=my-nginx"
You’ll then copy the new keys into the nginxsecret.yaml file and add it to the cluster.
apiVersion: "v1"
kind: "Secret"
metadata:
name: "nginxsecret"
namespace: "default"
type: kubernetes.io/tls
data:
tls.crt: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0..."
tls.key: "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0..."
After applying the secret, you’ll need to apply the service which is used by Kubernetes to connect ports with a label which is associated with the deployment. Note ‘labels‘ is ‘my-nginx‘ and in the deployment.yaml file, it has the same ‘labels‘ line. Traffic coming to this service will go to any pod (ingress-controller pod) with this label.
apiVersion: v1
kind: Service
metadata:
name: my-nginx
labels:
run: my-nginx
spec:
type: NodePort
ports:
- port: 8080
targetPort: 80
protocol: TCP
name: http
- port: 443
protocol: TCP
name: https
selector:
run: my-nginx
Then apply the following deployment.yaml which will pull the nginx image from docker.io.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 1
template:
metadata:
labels:
run: my-nginx
spec:
volumes:
- name: secret-volume
secret:
secretName: nginxsecret
- name: configmap-volume
configMap:
name: nginxconfigmap
containers:
- name: nginxhttps
image: bprashanth/nginxhttps:1.0
ports:
- containerPort: 443
- containerPort: 80
volumeMounts:
- mountPath: /etc/nginx/ssl
name: secret-volume
- mountPath: /etc/nginx/conf.d
name: configmap-volume
When you check the service, because it’s a NodePort, you’ll see both the service ports (8080 and 443) and the exposed ports (31201 and 31713). The exposed ports can be used to access the application by going to the Load Balancer url and adding the port.
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver NodePort 10.69.249.118 <none> 8080:32356/TCP 9h
kubernetes ClusterIP 10.69.0.1 <none> 443/TCP 9d
my-nginx NodePort 10.69.210.167 <none> 8080:31201/TCP,443:31713/TCP 27h
However that’s not an optimum process. You have to make sure users know what port is assigned and make sure the port is opened on your Load Balancer. With an Ingress Controller, you create a DNS CNAME that points to the Load Balancer and then apply this ingress.yaml route.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-nginx
namespace: default
annotations:
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: my-ingress.dev.internal.pri
http:
paths:
- backend:
service:
name: my-nginx
port:
number: 8080
path: /
pathType: Prefix
tls:
- hosts:
- my-ingress.dev.internal.pri
I created a my-ingress.dev.internal.pri DNS CNAME that points to bldr0cuomvip1.dev.internal.pri. When accessing https://my-ingress.dev.internal.pri, you’ll be directed to the my-ingress service which then transmits traffic to the application pod regardless of which worker node it resides on.
Let’s break this down just a little for clarity, in part because it didn’t click for me without some poking around and having a ping moment when looking at an old document I created for an Openshift cluster I was working on.
In the ingress.yaml file, the spec.rules.host and spec.tls.hosts lines are the DNS entries you created for the pod(s). The ingress controller looks for this service and transmits traffic to the configured service.
The spec.rules.http.backend.service.name is the name of the service this ingress route transmits traffic to. The service.port.number is the port listed in the pod service.
The path line is interesting. You can have multiple directories accessible by different DNS names by changing the path line. In general this is a single website so the / for the path is appropriate for a majority of cases.
The thing that is important is the annotations line. It has to point to the ingress controller. For the haproxy-ingress-controller, it’s as listed but you can verify by describing the pod .
kubectl describe pod haproxy-ingress-7bc69b8cc-wq2hc -n ingress-controller
...
Args:
--configmap=ingress-controller/haproxy-ingress
--ingress-class=haproxy
--sort-backends
...
In this case we see the passed argument of ingress-class = haproxy. This is the same as the annotations line and tells the ingress route which pod is load balancing traffic within the cluster.
Once applied, you can then go to https://my-ingress.dev.internal.pri and access the nginx startup page.
echoserver Project
This one is a little simpler but still can show us how to use an ingress route to access a pod.
All you need is a service.yaml file to know where to transmit traffic.
apiVersion: v1
kind: Service
metadata:
labels:
app: echoserver
name: echoserver
namespace: default
spec:
clusterIP: 10.69.249.118
clusterIPs:
- 10.69.249.118
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- nodePort: 32356
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: echoserver
sessionAffinity: None
type: NodePort
Then a deployment.yaml file to load the container.
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: echoserver
name: echoserver
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: echoserver
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: echoserver
spec:
containers:
- image: k8s.gcr.io/echoserver:1.3
imagePullPolicy: IfNotPresent
name: echoserver
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
For me, the problem was the example was a single line to create the ingress route but it wasn’t enough information to help me create the route. A lot of the problems with examples is they’re expecting cloud usage and you’ll have an AWS, GCE, or Azure load balancer. For on prem it seems to be less obvious in the examples which is why I’m doing it this way. It helps me and may help others.
Here is the ingress.yaml file I used to access the application. Remember you have to create a DNS CNAME for the access and you’ll need the port number from the service definition (8080).
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: echoserver
namespace: default
annotations:
kubernetes.io/ingress.class: haproxy
spec:
rules:
- host: echoserver.dev.internal.pri
http:
paths:
- backend:
service:
name: echoserver
port:
number: 8080
path: /
pathType: Prefix
tls:
- hosts:
- echoserver.dev.internal.pri
And with this ingress route, you have access to the echoserver pod. As I progress in loading tools and my llamas website, I’ll provide the ingress.yaml file so you can see how it’s done.
References
