Raspberry Pi Kubernetes Cluster: NFS Storage (Issue #5)

Welcome to the Raspberry Pi Kubernetes home lab series. A bite sized informative guide to help you provision a cluster from scratch. By now we should have our k8s cluster up and running; great job everyone.

Today's issue will walk you through installing and creating a Network File System (NFS) server so it can be used as our default cluster storage.

1. Requirements

• 1 k8s cluster
• 1 external hard drive

2. Locate the hard drive

Plug your hard drive into one of your slave nodes and SSH into it to find its location.

sudo lsblk -o UUID,NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL,MODEL

3. Format the hard drive with mkfs.ext3

We can see that our hard drive is located at /dev/sda2. Let's go ahead and format the hard drive's filesystem so it can be used by the Linux system. Other examples include /dev/sdb2.

sudo mkfs.ext3 -L k8s-volume /dev/sda2

4. Create a directory for the volume and mount it

This could be in any location. We are going to "namespace" it under k8s-mount and label it k8s-volume-1.

sudo mkdir -p /k8s-mount/k8s-volume-1
sudo mount /dev/sda2 /k8s-mount/k8s-volume-1 # mount disk

5. Install the NFS server

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install nfs-kernel-server -y

6. Start the server

sudo systemctl start nfs-server
sudo systemctl status nfs-server # verify it is running

7. Allocate permissions

In order to have access to your NFS server we need to create some read/write permissions.

# owner (read/write) group (read)
sudo find /k8s-mount/k8s-volume-1/ -type d -exec chmod 755 {} \;
sudo find /k8s-mount/k8s-volume-1/ -type f -exec chmod 644 {} \;
sudo find /k8s-mount/k8s-volume-1/ -type f -exec chmod 777 {} \;

8. Set up remote access

The line below will give permissions to all hosts in the network, so be careful!

sudo nano /etc/exports

/k8s-mount/k8s-volume-1/ *(rw,sync,no_subtree_check,insecure)

We can set the permissions to be restrictive per host. Let me know if you would like to see how this is done. In the meantime, here is a walk through of this setup and the official docs too.

9. Export the etc config

sudo exportfs -ra # export config
sudo exportfs -v  # see current config
showmount -e      # see what is being exported

10. Install the NFS client on all the other worker nodes

# ssh ubuntu@192.168.0.xx # WORKER_NODE
sudo apt install nfs-common -y

11. Create volume manifests

# system-volume.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: system-volume
  labels:
    name: system-volume
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-k8s-volume-1
  labels:
    type: local
spec:
  storageClassName: k8s-volume-1
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.26
    path: /k8s-mount/k8s-volume-1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-k8s-volume-1
spec:
  storageClassName: k8s-volume-1
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Ti

12. Apply the manifests

kubectl apply -k path-to-manifest.yaml
kubectl --namespace system-volume get pv,pvc

Bonus

This isn't a required section but more of an educational addendum. Since we have given global access to our drive, we can now access it anywhere from within the network.

1. Find nodes

nmap -sP 192.168.0.1/24 # list nodes
 
WORKER_NODE_1=192.168.0.24
WORKER_NODE_2=192.168.0.26 # hard drive lives here

2. Find the volume on the network

ssh ubuntu@$WORKER_NODE_1 # a worker node
 
sudo apt install nfs-common
 
showmount -e $WORKER_NODE_2 # see what is being exported
 
# output
Export list for 192.168.0.26:
/k8s-mount/k8s-volume-1 *

3. Access the hard drive

sudo mkdir -p /test
sudo mount -t nfs $WORKER_NODE_2:/k8s-mount/k8s-volume-1 /test/
ls -al /test/ # inspect drive

4. End of test

Clear the test area by removing the files created previously.

sudo umount /test
sudo rm -rf /test

We are now ready to start using our local storage. It is important that you know how to access the nodes independently and that you understand how to manipulate the NFS server across the network.

Leave a comment if you have any questions. We will be integrating with MetalLB's load balancer next. Happy coding.

First published in my LinkedIn newsletter, Built from Scratch.