Setting up a Raspberry Pi Kubernetes Cluster (with Blinkt! Strips that Show Number of Pods per Node!) Using k3sup
If, like me, you are interested in the worlds of both Raspberry Pi and Kubernetes you may have built or considered building a Raspberry Pi Kubernetes cluster (see here for just one of many examples). I built a three-unit cluster in early 2018 using Raspberry Pi 3 Model B+ boards and bootstrapped Kubernetes using an early version of Alex Ellis' guide and it was all pretty straightforward. By itself it's not a great thing to demo (in my case at my local Raspberry Jam for example) as there is no display so a nice improvement is to fit Pimoroni Blinkt! LED strips to the GPIO pins of each unit and then use the guide here to make the LEDs light up according to the number of pods that have been deployed. The Blinkt! improvement went fine and was working nicely—right up to the day when I decided to upgrade the Raspberry Pi OS from Raspbian Stretch to Raspbian Buster (which came out to support the new Raspberry Pi 4 model).
The first problem was that Docker wouldn't install on Buster but that was solved through a post by Alex Ellis. However I encountered other problems such as the swapfile not turning off between reboots and (critically) the Weave networking pods failing to start and I spent a lot of time messing around to no avail. The sensible option would have been to revert to Raspbian Stretch but by now I had the bit between my teeth and I wasn't giving up lightly. After even more messing about trying different configurations and getting nowhere I decided to follow Alex Ellis' advice and try k3s—a stripped-down version of Kubernetes from Rancher Labs (the k3s name is a twist on the often-used k8s abbreviation for Kubernetes). In fact, you can make things even easier by using Alex's k3sup tool to automate most of the process.
TL;DR: I had everything up-and-running on Raspbian Buster in next no time at all, including the Blinkt! LED strips displaying the number of pods on each node!
If you are looking to learn bare-metal Kubernetes installation then k3s/k3sup may not be for you. But if you just want to get a cluster configured with minimal fuss then it's just the ticket. As always there were a few twists and turns so here is a write-up of what worked for me, although I'm not documenting every single step because it's already well covered by Alex. I'm using a Windows 10 development machine so this write-up is from that perspective, however Alex's documentation is more Linux/macOS focused so everyone should be able to follow along.
Install k3s Using k3sup
I used three guides that together provided a complete picture for installing k3s on the Raspberry Pi platform using k3sup:
Use the first guide to (if necessary) build your cluster and then prepare each Pi. In addition to setting the GPU memory split, changing the hostname and changing the password for each Pi I also expanded the filesystem to use all of the SD card. You will need the IP addresses of your master and worker nodes so setting static IP addresses is a good way to go.
Follow the first guide up to and including Enable container features. I simply used sudo nano /boot/cmdline.txt to edit the file being careful not to add an extra line.
Use guides 2 and 3 to install k3s using k3sup. The key point to understand here is that you run k3sup from your development machine. As I'm running Windows 10 it was a case of grabbing the Windows binary from the k3sup releases page and copying it to my working folder for this project. On Windows bootstrapping the master node is simply a matter of opening a command prompt at your working folder and running this command, replacing $SERVER with the IP address of the master node:
1 |
k3sup install --ip $SERVER --user pi |
Bootstrapping the worker nodes is similarly straightforward:
1 |
k3sup join --ip $AGENT --server-ip $SERVER --user pi |
where $AGENT is the IP of the worker node and $SERVER the IP of the master node.
Communicating with the Cluster using kubectl
The k3s installation includes kubectl so from this point on it's just like working with any standard Kubernetes cluster. You'll obviously need kubectl installed on your development machine, and then you configure kubectl to talk to the cluster using whichever of the several techniques works best for you, using the kubeconfig file that is handily copied to the working folder on your workstation. In my case I chose simply to copy kubeconfig to ~\.kube\ (which had been previously created through working with Azure Kubernetes Service but you can create it yourself) and rename kubeconfig to config to end up with C:\Users\Graham\.kube\config. Running kubectl get nodes establishes that everything is (hopefully!) working correctly.
Configuring the Cluster so Pimoroni Blinkt! LED strips Indicate the Number of Pods per Node
The master guide to follow is here. With the Blinkt! LED strips installed you'll need to download the following files from the guide's repo to your working folder:
- kubernetes/blinkt-k8s-controller-rbac.yaml
- kubernetes/blinkt-k8s-controller-ds.yaml
You'll also need to create a manifest containing a deployment. Create a file in the working folder called deployment.yaml and copy the following:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
apiVersion: apps/v1 kind: Deployment metadata: name: busybox-httpd labels: blinkt: show spec: replicas: 5 selector: matchLabels: blinkt: show template: metadata: labels: blinkt: show spec: containers: - name: busybox-httpd image: hypriot/rpi-busybox-httpd |
Now run the following commands against the cluster, from a command prompt at the working folder:
- kubectl label node $NODE_NAME deviceType=blinkt (where $NODE_NAME is the name of each node with a Blinkt! strip)
- kubectl create -f blinkt-k8s-controller-rbac.yaml
- kubectl create -f blinkt-k8s-controller-ds.yaml
- kubectl apply -f deployment.yaml
At this point you should see five LEDs light up according to how Kubernetes has decided which nodes the five pods should run on. You can now open deployment.yaml in your favourite code editor and play around with the number of replicas, repeating the final command above after saving each change to the file. Watch in joy as pods are created with a green flash and destroyed with a red flash, and settle on a satisfying blue (which you can change in blinkt-k8s-controller-ds.yaml ) for running pods.
Final Thoughts
I've been thrilled with how easy k3sup makes installing k3s and even if you do want to experience the pain thrill of the kubeadm procedure on Raspberry Pi I would still recommend you check out Alex's posts mentioned here and others on his blog as they offer tremendous extra value and learning.
Cheers -- Graham