2022-09-30

Raspberry Pi Kubernetes Cluster: Start the Cluster (Issue #4)

Install Kubernetes on the Raspberry Pi cluster: containerd, kubeadm, the Flannel CNI, then initialise the master node and join the workers.

Start the Kubernetes cluster

Welcome to the Raspberry Pi Kubernetes home lab series. A bite sized informative guide to help you provision a cluster from scratch. By now we should have 3+ nodes running the latest version of Ubuntu and we should be able to SSH into them.

Today's issue will walk you through installing Kubernetes into your newly created cluster.

1. Requirements

At least 3 nodes with Ubuntu installed on them

The following 12 steps should be replicated on every node. We will set them up manually in this lab session, but will provide an automated way of setting up nodes via Ansible in a coming issue.

2. SSH into the node

ssh ubuntu@192.168.0.25

3. Set up hostnames and static IPs

Slave nodes need to provide a unique name to the master node upon registration.

# On each node
NODE_NAME=k8s-master # k8s-node-1 etc..
sudo hostnamectl set-hostname $NODE_NAME

4. Edit the hosts file so it maps IP addresses to hostnames

sudo nano /etc/hosts
192.168.0.25 k8s-master
192.168.0.26 k8s-node-1
192.168.0.24 k8s-node-2

5. Disable swap memory

Swap is space on the host disk that is used when the amount of physical RAM is full. Even though swap support has been added to Kubernetes, our Raspberry Pis have limited resources and we should account for such limitations.

sudo swapoff -a
sudo apt purge -y dphys-swapfile
# Test swap has been disabled
free -m

6. Edit the boot configuration

Append the following config to the boot file.

sudo nano /boot/firmware/cmdline.txt
ipv6.disable=1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

7. Set up a container runtime (CRI)

Kubernetes has moved away from Docker. There are a few options out there (Docker included). We will be setting up containerd as our CRI.

sudo apt update -y && sudo apt dist-upgrade -y
sudo apt install -y containerd containernetworking-plugins

8. Create a config for containerd

sudo mkdir /etc/containerd
cat <<EOF | sudo tee /etc/containerd/config.toml
version = 2
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    [plugins."io.containerd.grpc.v1.cri".containerd]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
 
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true
EOF

9. Forward IPv4 and set up IP tables

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
 
sudo modprobe overlay
sudo modprobe br_netfilter
 
# sysctl params required by setup, params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
 
# Apply sysctl params without reboot
sudo sysctl --system

10. Install kubeadm

Update the apt package index and install packages needed to use the Kubernetes apt repository:

sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl

Download the Google Cloud public signing key:

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

Add the Kubernetes apt repository:

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

Update the apt package index, install kubelet, kubeadm and kubectl, and pin their version:

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

11. Install a Container Network Interface (CNI)

A CNI is required to implement the Kubernetes network model. We are going to install Flannel in this lab as it is compatible with MetalLB. Be sure to check out different networking models too, such as Calico.

wget https://github.com/flannel-io/flannel/releases/download/v0.19.2/flanneld-arm64
sudo chmod +x flanneld-arm64
sudo cp flanneld-arm64 /usr/local/bin/flanneld
sudo mkdir -p /var/lib/k8s/flannel/networks

We will install Flannel and apply its configuration once the master node is running.

12. Reboot (optional)

Reboot the nodes and start fresh. It will ensure that the boot file from step 6 is loaded in.

sudo reboot

Start the cluster

SSH into the master node and run:

sudo kubeadm config images pull
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --token-ttl=0

Save the token and discovery token provided by the initialisation. You can always generate a new token later by doing:

KUBE_TOKEN=$(kubeadm token generate)
kubeadm token create $KUBE_TOKEN --print-join-command --ttl=0

Create the kube config

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Node verification

kubectl get nodes
NAME       STATUS   ROLES  AGE VERSION
k8s-master NotReady master 55m v1.25.1

For the node to become ready we need to install our CNI from step 11.

Apply the CNI config

Assuming we are connected to our master node from the previous section, download Flannel's configuration:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Inspect the YAML configuration and replace the backend. More info is in the Flannel backend documentation.

sed -i 's/vxlan/host-gw/' kube-flannel.yml

Apply the config:

kubectl create -f kube-flannel.yml

Add slave nodes to the cluster

SSH into the slave nodes (k8s-node-1, k8s-node-2, ...) and run:

K8S_MASTER_IP=192.168.0.25
sudo kubeadm join $K8S_MASTER_IP:6443 \
--token <token here> \
--discovery-token-ca-cert-hash sha256:<hash value here>
# Output:
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

We are now ready to start inspecting and extending our cluster.

Leave a comment if you have any questions. We will be automating this setup next. Happy coding.

First published in my LinkedIn newsletter, Built from Scratch.

ShareX LinkedIn