Kubernetes High-Availability Cluster Setup with kubeadm
Build a highly available Kubernetes control plane with kubeadm and a load balancer in this preserved historical installation guide.
Published · Republished on Medium
Historical article (December 2020): this page preserves an old lab procedure. Do not use its package repositories, container runtime setup, remote manifests, CNI, ingress resources, or version assumptions as a current installation guide. Build a new plan from the current kubeadm documentation and a currently supported Kubernetes release.
Kubernetes?
Kubernetes is an open-source platform used for managing containerized application workloads, as well as providing declarative configuration and automation. Kubernetes is in a large and fast-growing ecosystem. Kubernetes services, support, and tools are widely available.
Kubernetes provides container-centered environmental management. Kubernetes manages to compute networking and storage infrastructure. This feature then makes the concept of Platform as a Service (PaaS) simpler, flexible, equipped with Infrastructure as a Service (IaaS).
Historical HA Design
The intended design places one TCP load balancer in front of three control-plane nodes. One correction is essential: the load balancer address used as kubeadm's controlPlaneEndpoint is not bootstrap-only and must not be turned off after initialization. Nodes, administrators, and components need a stable API endpoint. A single load-balancer VM is itself a single point of failure, so this lab has multiple control-plane nodes but does not provide end-to-end high availability.
This article actually uses six VMs:
- 1 VM load balancer, as control plane or proxy in the Kubernetes cluster.
- 3 VM Master, as control manager in the Kubernetes cluster.
- 2 VM workers, where pods or services are stored
Preserved 2020 Procedure
The commands below are retained for historical context. Several are obsolete or unsafe to copy today: apt-key is deprecated, the Google-hosted Kubernetes apt repository shown here is retired, Docker Engine is not the assumed kubeadm runtime, remote manifests are unpinned, and the old Weave URL should not be executed.
Host Preparation
- set hosts on all node
config
vi /etc/hosts
...
10.61.61.10 k8s-LB
10.61.61.11 k8s-master1
10.61.61.12 k8s-master2
10.61.61.13 k8s-master3
10.61.61.14 k8s-worker1
10.61.61.15 k8s-worker2
...- setup ssh passwordless authentication from node master1 to another master
config
sudo -i
ssh-keygen
cat /etc/hosts | grep master | awk {'print $2'} > target.txt
for node in $(cat target.txt); do ssh-copy-id root@$node; done- Testing ssh passwordless authentication
for node in $(cat target.txt); do ssh root@$node hostname; done
config
''output
...
k8s-master1
k8s-master2
k8s-master3
...- Update and upgrade package on all nodes
sudo apt -y update; sudo apt -y upgrade; sudo apt -y update;
- Install docker.io on all node, except node LB
config
sudo apt install -y docker.io; sudo docker version
sudo systemctl enable docker
sudo systemctl start docker
sudo systemctl status docker- Install kubelet, kubeadm, kubectl on all nodes except the load balancer (historical repository)
Archive warning: do not run the following repository commands today. Current Kubernetes documentation uses
pkgs.k8s.iorepositories that are scoped by minor version and requires version-skew planning for kubeadm, kubelet, kubectl, and the control plane.
sudo apt install -y apt-transport-https; curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
config
cat <<EOF > kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOFconfig
sudo mv kubernetes.list /etc/apt/sources.list.d/kubernetes.list
sudo apt update; sudo apt install -y kubectl kubelet kubeadm- Set off swap on all node, except node LB
config
sudo swapon -s
sudo swapoff /dev/xxx
sudo swapon -s- Install and configure haproxy on node LB
config
sudo apt update; sudo apt upgrade -y; sudo apt install haproxy -y
sudo vim /etc/haproxy/haproxy.cfg
...
frontend kubernetes
bind 10.61.61.10:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodesconfig
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master1 10.61.61.11:6443 check fall 3 rise 2
server k8s-master2 10.61.61.12:6443 check fall 3 rise 2
server k8s-master3 10.61.61.13:6443 check fall 3 rise 2config
frontend https_frontend_kubernetes
bind 10.61.61.10:443
option tcplog
mode tcp
default_backend backend_k8s_nodesconfig
backend backend_k8s_nodes
mode tcp
balance roundrobin
option tcp-check
server k8s-master1 10.61.61.11:6443 check fall 3 rise 2
server k8s-master2 10.61.61.12:6443 check fall 3 rise 2
server k8s-master3 10.61.61.13:6443 check fall 3 rise 2
...- Verification the configure haproxy and then restart the service
config
haproxy -c -V -f /etc/haproxy/haproxy.cfg
''output
...
Configuration file is valid
...sudo systemctl restart haproxy
- Verification connection between master and node load balancer
config
nc -v 10.61.61.10 6443
''output
...
Connection to 10.61.61.10 6443 port [tcp/*] succeeded!
...- Initialization on master1
kubeadm init --pod-network-cidr=10.244.X.0/16 --control-plane-endpoint "IP_LOADBALANCER:6443" --upload-certs
- Install the historical CNI from master1
config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configThe original post downloaded and applied a mutable Weave manifest from the internet. That command has been removed because it is not a safe or current CNI installation path. Select a maintained CNI compatible with the chosen Kubernetes version, pin and review its manifest or chart, and verify network policy and upgrade ownership.
config
kubectl get pod -n kube-system -w
''note: just wait until all pods already up.- Joining master2 & master3 to cluster node control-plane
config
kubeadm join IP_LOADBALANCER:6443 --token [TOKEN] \
--discovery-token-ca-cert-hash [TOKEN-ca-cert-hash] \
--control-plane --certificate-key [certificate-key]- Verification on node master1
kubectl get nodes
- Execution on node master2 & master3 and verification
config
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/configkubectl get nodes
''note: It is executed so that you can run kubectl commands to display a list of nodes, or other kubectl commands on master2 & master3
- Create three different deployments
git clone https://github.com/riowiraldhani/kubernetes
config
cd kubernetes/
lsconfig
./createdeployments.sh
''output:
...
''list deployment
...- Expose deployment with type NodePort
config
- ./exposedeployments.sh
''output:
...
''list service with nodePort
...config
- verification
curl localhost:30606
...
Hello, world!
Version: 2.0.0
Hostname: helloapp-2-5ccf4846b5-pnwss
...- Create ingress for three service that we have created
kubectl apply -f helloapp-ingress.yaml
kubectl get ing --all-namespaces
- Create service ingress controller
config
kubectl apply -f ingress-controller.yaml
kubectl get svc -n ingress-nginx- Edit service ingress controller
config
kubectl edit svc -n ingress-nginx ingress-nginx-controller
...
spec:
clusterIP: 10.101.62.144
externalIPs:
- IP_LOADBALANCER
...
''note: add externalIPsconfig
kubectl get ingress helloapp-ingress
''note: now ingress helloapp-ingress have Address- Verification service ingress
config
kubectl get ingress --all-namespaces
kubectl get svc -n ingress-nginxconfig
''note: now, you have service with hostname. But, you need mapping the hostname with you'r ip master. example;
...
10.61.61.11 k8s-master1 helloword-v1.info
...config
curl helloword-v1.info:[NodePort]
''output
...
Hello, world!
Version: 1.0.0
Hostname: helloapp-1-759f7597c5-8sf2r
...- Access service with IP Node Loadbalancer
''note: if you want expose or access this service with IP node load balancer you need add config on haproxy.
''on node k8s-LB
config
sudo vi /etc/haproxy/haproxy.cfg
...
frontend nginx
bind 10.61.61.10:5000
mode http
default_backend backend_nginxconfig
backend backend_nginx
balance roundrobin
server k8s1 10.61.61.11:30606
server k8s2 10.61.61.12:30606
server k8s3 10.61.61.13:30606config
frontend nginx-1
bind 10.61.61.10:5001
mode http
default_backend backend_nginx-1config
backend backend_nginx-1
balance roundrobin
server k8s1 10.61.61.11:30734
server k8s2 10.61.61.12:30734
server k8s3 10.61.61.13:30734
...config
haproxy -c -V -f /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy''note: port 30606 & 30734 is port NodePort, you can get this with command kubectl get svc and look at column PORT(S)
- Verification
''on node k8s-LB and other node
config
curl 10.61.61.10:5000
...
Hello, world!
Version: 1.0.0
Hostname: helloapp-1-759f7597c5-8sf2r
...config
curl 10.61.61.10:5001
...
<html><body><h1>It works!</h1></body></html>
...What a Current Design Must Revisit
A current kubeadm HA design must choose stacked or external etcd, provide a stable and highly available TCP endpoint for the API server, match the controlPlaneEndpoint, use supported package repositories and a CRI-compatible runtime, pin a supported CNI, and plan certificate distribution, etcd backup and restore, upgrades, node failure, and load-balancer failure.
The application-exposure section is also historical. Editing Services interactively, assigning externalIPs manually, and forwarding arbitrary NodePorts through the same single HAProxy instance do not establish a resilient ingress architecture. Use a maintained ingress or Gateway implementation and explicitly design address advertisement, TLS, health checks, network policy, and failure behavior.
Before calling a cluster highly available, test loss of each control-plane node, an etcd member, the API load-balancer instance, and the network path. This archived topology fails the load-balancer test by design.