Platform

GitOps-managed k3s cluster running data + ML platform services on batzbakserver.

Structure

• bootstrap/ — one-time manual ArgoCD install + root Application
• apps/ — ArgoCD Application manifests, one per service
• charts/ — Helm values per service
• docs/ — architecture notes and session logs

Cluster

Single-node k3s v1.34.6 running in a Proxmox VM on batzbakserver.

Prerequisites for rebuild

To rebuild this cluster from scratch, you'll need:

• A Proxmox VE host (or any Linux host capable of running a VM)
• The sealed-secrets-master-key-BACKUP.yaml file — irreplaceable, stored OUTSIDE this repo. Without it, every committed sealed secret becomes permanently undecryptable
• A password manager with the entries listed in docs/rebuild.md
• DNS for *.lab.batzbak.top (or your equivalent) resolvable from the cluster, plus a Cloudflare API token for cert-manager's DNS-01 flow
• A devbox (any Linux workstation) with kubectl, helm, argocd, kubeseal, and mc CLIs installed

Full from-scratch rebuild runbook: docs/rebuild.md.

Services

To be populated as sessions progress:

• k3s cluster bootstrapped
• ArgoCD + sealed-secrets (session 1)
• ingress-nginx + cert-manager (session 2)
• CloudNativePG + MinIO (session 3)
• Tenant data tier (postgres-tenants) + wedding-site migration (session 5)
• Stakater Reloader for cross-repo Secret rollouts (session 6)
• MLflow (session 7)
• Airflow (session 8) — Dagster originally planned, switched after a cross-orchestrator comparison; see session-2026-05-13.md
• Lakekeeper Iceberg REST Catalog (session 9) — API + read path validated; write path validated in the session-9 third checkpoint ("session 10") by fixing remote-signing at the source (warehouse remote-signing-enabled: false) + PyIceberg/FsspecFileIO. See session-2026-05-22.md.
• insurance-retention MLOps showcase (2026-05-24/25): first real ML workload. A 3-model decision bundle trained by an Airflow DAG (KubernetesPodOperator) that a GHCR image-sensor auto-triggers on each new build via a data-aware Asset (code-driven CT, with image+commit lineage), versioned and gated-promoted in the MLflow registry, and served by an ArgoCD-managed FastAPI Deployment at https://insurance-retention.lab.batzbak.top. Chart in charts/insurance-retention/, DAG in airflow/dags/, recreation in docs/rebuild.md Step 12. Workload repo: https://github.com/SDR3078/insurance-retention
• Iceberg data plane — medallion bronze.train / bronze.score → gold.selections via PyIceberg DAGs (platform-side notes sessions 11–13, reconstructed 2026-06-08). Supersedes the planned "Trino + first Iceberg DAG": the write path was validated with PyIceberg, and Trino (1–2Gi always-on) was rejected for this single-node ML node — it returns only if an analytics-SQL consumer appears.
• Model registry + gated promotion + in-cluster tuning lifecycle (MLflow aliases, NV non-regression gate, tune → promote-params → train)
• Data quality + scheduling (bronze DQ is minimal/structural today)
• kube-prometheus-stack (next — the platform now runs blind; overdue)
• Pipeline & model observability
• LiteLLM + Langfuse + pgvector
• Portfolio chatbot as cluster app (2026-06) — Next.js web + FastAPI chat api, deployed on the wedding-site bootstrap+workload pattern from a private cross-repo chart. Hardened with the cluster's first least-privilege ArgoCD AppProject, a read-only GITHUB_TOKEN + Cloudflare Turnstile bot-check (SealedSecrets, resealed via kubeseal --merge-into), and a DeepInfra chat backend. Platform-side lessons in the course notes (session-2026-07-01); app-side in the portfolio-site course folder.
• QuestSync CalDAV bridge as cluster app (2026-07-01) — public Habitica⇄Radicale façade at https://tasks.batzbak.top. Stateless (no app secret — each user brings their own Habitica creds), public GHCR image, exposed via the EXTERNAL cloudflared tunnel → node ingress-nginx (a 4-expert panel chose reusing the LXC tunnel over a new in-cluster one). Hardened to portfolio-site parity: least-privilege AppProject, restricted PSA + seccomp, default-deny NetworkPolicy, pinned :sha image, dedicated cert. The public app repo is kept generic; operator glue (ns/cert/ingress) lives in charts/questsync/. Workload repo: https://github.com/SDR3078/questsync