Skip to content

Project Context for AI Agents

This file contains critical rules and patterns that AI agents must follow when implementing code in this project. Focus on unobvious details that agents might otherwise miss.

Technology Stack & Versions

Infrastructure as Code

  • Terraform >= 1.7.4 — local backend at terraform/main/state/terraform.tfstate
  • Providers (pinned in terraform/main/providers.tf):
  • scott-the-programmer/minikube ~> 0.6.0
  • hashicorp/kubernetes ~> 2.26.0
  • hashicorp/helm ~> 2.12.1
  • oboukili/argocd 6.0.3

Cluster & GitOps

  • Minikube — Kubernetes v1.33.4, driver docker (switched from qemu2/socket_vmnet per sprint-change-proposal-2026-06-20 — IT firewall blocks VM-driver SSH). Host access needs sudo minikube tunnel -p data + /etc/hosts → 127.0.0.1 argocd.data airflow.data (the docker tunnel binds localhost, not the node IP). See docs/runbooks/clean-state-bring-up.md.
  • ArgoCD — apps & projects provisioned via Terraform (modules/k8s/argocd/{project,application,server}); apps use an automated syncPolicy (prune + self-heal) so they self-reconcile.

Data Platform Workloads

  • Apache Airflow 3.2.2 — official Apache Helm chart 1.22.0, wrapped at helms/airflow (config under the airflow: alias in values.yaml). CeleryExecutor + Redis; components: api-server, scheduler, standalone dag-processor, triggerer, worker. SimpleAuthManager admin; DAGs via gitSync (main, airflow subPath). Secrets are Terraform-managed k8s Secrets.
  • PostgreSQL 14.2.4 — Bitnami Helm chart (db mini_data_platform, user mini); images sourced from bitnamilegacy/*.

Python (DAGs)

  • Python >=3.12,<3.13, managed with Poetry
  • black ^24.2.0 (line-length 120), isort (black profile), flake8 ^7.0.0 (+ flake8-pyproject), mypy ^1.8.0 (strict), pytest ^8.0.2 + pytest-cov

Docs & CI

  • MkDocs → GitHub Pages (.github/workflows/ci-mkdocs.yml)
  • terraform-docs + helm-docs auto-inject into readme.md/README.md
  • pre-commit orchestrates per-area hooks (terraform, python, helms)

Critical Implementation Rules

Terraform Conventions

  • Numbered file ordering — files within a module/main are prefixed to convey order: 00-variables.tf, 01-*.tf … 99-outputs.tf. Follow this when adding files.
  • Modular structure — reusable infra lives under terraform/modules/k8s/* (minicluster, namespace, argocd/{project,application,server}). Compose in terraform/main/, don't inline resources there.
  • Local state — backend is local (terraform/main/state/). Do not assume remote state.
  • Provider config flows from the cluster module — kubernetes/helm providers are wired from module.cluster outputs in 01-cluster.tf.

Auto-Generated Docs — DO NOT HAND-EDIT

  • Terraform readme.md and Helm README.md content between <!-- BEGIN_TF_DOCS -->/<!-- END_TF_DOCS --> markers is generated by terraform-docs / helm-docs via scripts/*_docs.sh. Edit .tf variable descriptions or README.md.gotmpl instead; regeneration runs in pre-commit.

Airflow DAG Authoring (Airflow 3 / airflow.sdk)

  • All DAGs MUST subclass ChesterDag (mini_dags/chester/dag.py), not raw DAG. It enforces defaults: catchup=False, retries=1, no failure/retry emails, retry_delay=1m. See mini_dags/dags/example.py for the canonical pattern.
  • DAGs use the context-manager form (with ChesterDag(...) as dag:).
  • Airflow 3 SDK imports: from airflow.sdk import DAG; ScheduleArg comes from airflow.sdk.definitions.dag (not re-exported at the airflow.sdk top level in 3.2.2). Use the schedule= argument (no schedule_interval/timetable).

Python Code Quality

  • mypy is strict: disallow_untyped_defs + disallow_incomplete_defs — every function needs full type annotations. airflow.* imports are exempted from missing-import errors (configured in pyproject.toml overrides).
  • black line-length 120 (not default 88); isort uses the black profile.
  • flake8 ignores E203,E266,E501,B008,W503, max-cognitive-complexity 12.

Workflow

  • pre-commit gates everything — it runs per-directory sub-hooks (terraform, python via cd airflow, helms) plus doc regeneration. Run before committing.
  • Commit messages follow a gitmoji + type convention, e.g. feat: :wrench: Deploy Postgres instance.

Platform Operational Gotchas (Airflow 3 / k8s 1.33 — landed)

  • helms/airflow is the OFFICIAL Apache chart 1.22.0 (Airflow 3.2.2) wrapped under the airflow: alias — no longer a custom mini chart. All config lives in values.yaml.
  • Host access (docker driver): sudo minikube tunnel -p data + /etc/hosts → 127.0.0.1 (NOT the node IP). The tunnel must be restarted after any cluster recreate.
  • Clean bring-up is two-phase (argocd-provider bootstrap): -target=module.cluster,module.argocd first (tunnel), then full terraform apply. See docs/runbooks/clean-state-bring-up.md.
  • SimpleAuth admin password is auto-generated/ephemeral — read it from the api-server logs.
  • DB migrate runs as an ArgoCD hook: Sync job (useHelmHooks: false); never re-enable Helm hooks.

Usage Guidelines

For AI Agents: - Read this file before implementing any code. - Follow ALL rules exactly as documented; when in doubt, prefer the more restrictive option. - Update this file if new patterns emerge.

For Humans: - Keep this file lean and focused on agent needs. - Update when the technology stack changes (e.g. after the Airflow 3.0 / k8s 1.33 upgrades land). - Review periodically and remove rules that become obvious over time.

Last Updated: 2026-06-21