Mini Data Platform

This project aims to showcase the value and capabilities to deploy a local K8s cluster with the minimum Data Platform stack we would start in any company. This means having K8s with ArgoCD & Airflow.

Terraform setup

This section has been generated with the terraform-docs pre-commit repo.

Setup

For the mini-data-platform deployment we will require:

minikube to deploy the K8s cluster
tfenv to manage our Terraform version
colima for running docker
qemu driver for K8S VM creation
OpenLens to interact with K8s
trivy for security checks

setup.sh

brew install docker
brew install colima
brew install minikube
brew install trivy
# Configure Terraform
brew install tfenv
tfenv install latest
tfenv use latest
# extras
brew install --cask openlens

Qemu Networking

Info

The QEMU driver has two networking options: socket_vmnet and builtin. socket_vmnet will give you full minikube networking functionality, such as the service and tunnel commands. See: docs

qemu_setup.sh

minikube start --driver=qemu --download-only
brew install socket_vmnet
brew tap homebrew/services
HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet

Configure local network

Since we take advantage of the ingress add-on, we must configure our local hosts and resolvers to enable us to browser our applications using our own local domain.

Tip

The default configuration exposes all the endpoints into the *.data domain. (i.e. argocd.data & airflow.data).

Once the minikube cluster is deployed via terraform, we will obtain its ip with the following command:

$ minikube ip -p data
192.168.105.4

Then, we will provide the new resolver in addition of updating the hosts file.

/etc/resolver/minikube-data

domain data
nameserver 192.168.105.4
search_order 1
timeout 5

/etc/hosts

192.168.105.4 argocd.data
192.168.105.4 airflow.data

Requirements

Name	Version
terraform	>= 1.7.4
argocd	6.0.3
helm	~> 2.12.1
kubernetes	~> 2.26.0
minikube	~> 0.3.10

Providers

No providers.

Modules

Name	Source	Version
application	../modules/k8s/argocd/application	n/a
application_db	../modules/k8s/argocd/application	n/a
argocd	../modules/k8s/argocd/server	n/a
cluster	../modules/k8s/minicluster	n/a
namespace	../modules/k8s/namespace	n/a
project	../modules/k8s/argocd/project	n/a

Resources

No resources.

Inputs

No inputs.

Outputs

No outputs.