-
Notifications
You must be signed in to change notification settings - Fork 0
Development Cluster
We use two types of development clusters, the Digital Twin Development Cluster
and the Kind cluster
.
One of the cluster types uses Vagrant virtual machines to instantiate all the machines and test OS images and networking configurations.We call it the Digital twin developer cluster
as it copies the actual physical setup of the cluster.
The second type, the Kind cluster
implements a ready-to-go Kubernetes (K8s) cluster to test deployments and K8s-only configs.
The Development Cluster is a way to ease the development and test NIployments-revamp effectively. It uses Vagrant to standardize the deployment of VMs across platforms and providers.
- Install Vagrant
- Install a virtualization provider. You can use Virtualbox (Windows/Linux/macOS) or libvirt (Linux only, but more performant than Virtualbox).
- Edit dev-cluster.yaml to suit your pc configuration. You can adjust RAM for each node, the number of nodes, and networking (important if you are using libvirt)
- Run the cluster with
vagrant up
(take a coffee, it can take a while for the first time). In order to stop it, you can runvagrant halt
You can SSH to each node in the cluster, including the router, doing vagrant ssh cluster<n>
, being n the node number (So if you have 3 nodes, you can access cluster1, cluster2 and cluster3). In order to access the router you can do vagrant ssh router
.
In order to delete the cluster you can run vagrant destroy -f
.
To run the deploy-playbook.yaml
file you need jmespath
:
pip install jmespath
If you wish to change the default development cluster configuration, you must copy the dev-cluster.yaml
into local-dev-cluster.yaml
and make your changes there.
Vagrant requires that eth0 on the guest VM is configured for NAT for SSH (it uses SSH, for provisioning and ease of access to the VM itself).
Right now, the router runs on Debian 10, and the nodes run on Ubuntu 22.04 (but temporarily until we decide a suitable OS for the cluster)
All development and customization to the development cluster is done on the Vagrantfile in Ruby
Caution
There is a script, setup-kind-cluster.sh
, stored under dev
, which automates this whole process. Follow the instructions bellow if you wish to understand each step.
Important
You'll need docker-mac-net-connect installed and running in order to access the exposed services from your local machine.
Despite kind
creating a network automatically, we create the network used by the cluster manually so we can control the IP's of the services running on the cluster.
To do so run:
docker network create kind \
--subnet=172.28.0.0/16 \
--gateway 172.28.0.1 \
--opt "com.docker.network.bridge.enable_ip_masquerade"="true" \
--opt "com.docker.network.driver.mtu"="1500"
This config file allows the kubelet (running containerd) inside the containerised kind nodes to pull from registries with unverified TLS
certificates. Check out more in this StackOverflow post.
kind create cluster --config dev/test-cluster.kind.yaml
Despite not being used in the actual cluster, we use MetalLB to ease the networking config when using the Kind development cluster, making services accessible to the local machine and the container nodes running the K8s nodes.
Warning
More specifically, skipping this step will cause the services to not have an assigned IP address, being unavailable from your computer.
kubectl apply -f https://github.com/metallb/metallb/raw/main/config/manifests/metallb-native.yaml
kubectl wait --namespace metallb-system \
--for=condition=ready pod \
--selector=app=metallb \
--timeout=120s
echo "\
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: example
namespace: metallb-system
spec:
addresses:
- 172.28.255.200-172.28.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: empty
namespace: metallb-system" | \
kubectl apply -f -
And from here, you should be ready to go and install your services. You might find it interesting to add an Image Registry; check the wiki page about that.
๐๐๐๐๐๐๐๐๐๐๐ (โธโธโธ>๏น<โธโธโธ)