Skip to content

ROADMAP.md

Latest commit

 

History

History
91 lines (74 loc) · 4.58 KB

ROADMAP.md

File metadata and controls

91 lines (74 loc) · 4.58 KB
Raw

ObservaKit Roadmap

Shipped (v0.1.x)

  • Freshness Monitor (configurable SLA thresholds, Prometheus metrics)
  • Volume Anomaly Detection (Z-score, rolling average, min-history guard)
  • Quality Checks (Soda Core, Great Expectations, custom SQL)
  • Schema Drift Detector (column add/remove/type-change)
  • Pipeline Health (Airflow + Prefect REST API)
  • FinOps Tracker (Snowflake credits, BigQuery bytes)
  • Native dbt Integration (run_results.json + manifest.json, no package needed)
  • Column Profiling (null %, distinct count, min/max/mean)
  • Cross-Table Consistency Checks (row_count_match, sum_match)
  • Lineage-aware Alerts (downstream impact in alert message)
  • Alert Deduplication (60-minute window per table+type)
  • Alert Suppression (planned maintenance windows)
  • Slack + Email alerts with routing rules
  • Embedded React Dashboard (/ui)
  • CLI (observakit status, check, profile, suppress)
  • PostgreSQL, BigQuery, Snowflake connectors
  • Docker Compose (full stack + lite mode)
  • Distribution Drift Monitor (categorical value-share shifts, numeric mean shifts, null % drift)
  • Data Contracts (YAML-defined schema + business rule enforcement, violation alerts)
  • MySQL / MariaDB connector
  • Amazon Redshift connector (with SVV_COLUMNS + IAM auth support)
  • Discord alert channel (rich embeds)
  • Generic outgoing Webhook alerts (HMAC-signed, works with PagerDuty, Opsgenie, n8n)
  • /healthz endpoint (Kubernetes liveness/readiness probe)
  • DuckDB / Databricks / Trino connectors
  • PagerDuty native integration (Events API v2)
  • Microsoft Teams alert channel (Adaptive Cards)
  • observakit init interactive setup wizard
  • observakit validate-config & observakit diff CLI tools
  • Automatic Query Retries (tenacity-based resiliency for all warehouses)
  • Centralized Alert Auditing (AlertLog table)
  • Slack Block Kit support (richer notifications with severity colour-strips)

v0.2.0 — Storage & Advanced Monitoring

  • Delta Lake support (via delta-rs / deltalake Python library; S3, GCS, ADLS, local)
  • Backfill detection (timestamp-distribution analysis to suppress false anomaly alerts during historical re-ingestion)
  • Late-arriving data detector (cron-aware arrival windows with grace periods; fires alert when expected batch is missing)
  • Scheduled metadata purge (configurable per-table retention; daily purge job with audit log)

v0.3.0 — Incident Management & Analytics

  • Opsgenie integration
  • Incident timeline — link related alerts into a single incident
  • Alert recovery notifications (auto-send "resolved" when checks pass again)
  • Weekly digest email / Slack summary (scheduled health report)
  • On-call rotation support (route alerts based on time-of-day)

v0.4.0 — UI & Developer Experience

  • Enhanced React dashboard with per-table drill-down pages
  • Contract management UI (create/edit contracts without editing YAML)
  • Distribution trend charts (visualise value shares over time)
  • Check history timeline (visualise pass/fail streaks)
  • Dark mode
  • Python SDK (pip install observakit) for programmatic access
  • Pre-commit hook — run quality checks in CI before merging

v0.5.0 — Advanced Quality

  • Column-level lineage (track which columns feed which downstream columns)
  • Data catalog integration (Amundsen, DataHub, Atlan metadata linking)
  • Partition freshness monitoring (BigQuery/Snowflake partition-level SLAs)
  • Regex / format validation checks (emails, phone numbers, UUIDs)
  • Cross-column consistency (e.g. end_date >= start_date)
  • Row-level sampling for failed checks (show actual failing rows in alert)
  • Materialized view staleness tracking

Future / Considering

  • dbt Cloud API integration (replace local artifact parsing for cloud-hosted dbt)
  • Kafka / streaming source freshness monitoring
  • Multi-warehouse support (monitor multiple warehouses in one instance)
  • RBAC (read-only vs admin roles)
  • Multi-tenancy (multiple projects in one instance with isolation)
  • Cloud-hosted SaaS version — maintained by WillowVibe (for teams that want zero ops)
  • Anomaly explanation (LLM-powered root cause suggestions)

How to influence the roadmap

Open a GitHub issue with the label roadmap or vote on existing issues with a 👍. Features with the most votes move up the priority list.

Contributions welcome — if you implement any of the above, we'd love a PR!