Skip to content

Mini Data Platform

This project aims to showcase the value and capabilities to deploy a local K8s cluster with the minimum Data Platform stack we would start in any company. This means having K8s with ArgoCD & Airflow.

Terraform setup

This section has been generated with the terraform-docs pre-commit repo.

Setup

For the mini-data-platform deployment we will require:

  • minikube to deploy the K8s cluster
  • tfenv to manage our Terraform version
  • colima for running docker
  • docker driver for the K8s node (qemu2/vfkit VM SSH is blocked by the corporate firewall on this host)
  • OpenLens to interact with K8s
  • trivy for security checks
setup.sh
brew install docker
brew install colima
brew install minikube
brew install trivy
# Configure Terraform
brew install tfenv
tfenv install latest
tfenv use latest
# extras
brew install --cask openlens

Docker-driver networking (tunnel + /etc/hosts)

Info

The cluster runs on the docker driver (the qemu2/vfkit VM drivers are blocked by the corporate firewall on this host). On the docker driver the node IP (192.168.49.2) is not host-routable on macOS — minikube tunnel binds the ingress to 127.0.0.1 instead.

To reach argocd.data / airflow.data from the host, run the tunnel in a real terminal (it needs sudo) and point the *.data hostnames at 127.0.0.1:

tunnel.sh
sudo minikube tunnel -p data   # keep running; restart after any cluster recreate
/etc/hosts
127.0.0.1 argocd.data airflow.data

Tip

The argocd Terraform provider also reaches ArgoCD via argocd.data:443, so the tunnel must be up for terraform apply to manage ArgoCD Applications. The full clean bring-up sequence (two-phase apply + tunnel timing) is documented in docs/runbooks/clean-state-bring-up.md.

Requirements

Name Version
terraform >= 1.7.4
argocd 6.0.3
helm ~> 2.12.1
kubernetes ~> 2.26.0
minikube ~> 0.6.0
random ~> 3.6

Providers

Name Version
kubernetes 2.26.0
random 3.9.0

Modules

Name Source Version
application ../modules/k8s/argocd/application n/a
application_db ../modules/k8s/argocd/application n/a
argocd ../modules/k8s/argocd/server n/a
cluster ../modules/k8s/minicluster n/a
namespace ../modules/k8s/namespace n/a
project ../modules/k8s/argocd/project n/a

Resources

Name Type
kubernetes_secret.airflow_config_credentials resource
kubernetes_secret.airflow_metadata resource
kubernetes_secret.airflow_result_backend resource
random_bytes.fernet_key resource
random_password.admin resource
random_password.api_secret_key resource
random_password.jwt_secret resource
random_password.postgres resource

Inputs

No inputs.

Outputs

No outputs.