diff --git a/README.md b/README.md index bb0e731..2139d30 100644 --- a/README.md +++ b/README.md @@ -1,159 +1,120 @@ # Marketing Performance Analysis -End‑to‑end analytics stack for the Maven Fuzzy Factory e‑commerce dataset (teddy bears). This repo ingests CSVs with DLT, transforms with dbt, orchestrates with Dagster, and serves KPIs for a Metabase dashboard. +End-to-end analytics stack to analyze marketing performance for the **Maven Fuzzy Factory** e-commerce dataset. The project ingests CSVs with **DLT**, transforms with **dbt**, orchestrates with **Dagster**, and surfaces **KPIs** consumable in **Metabase**. -## What This Project Answers -- Trend of website sessions and orders -- Session‑to‑order conversion rate over time -- Best performing marketing channels +## Business Questions Covered +- Trends in sessions and orders +- Session-to-order conversion rate +- Performance by acquisition channel (utm_source) +- Revenue, gross margin, and refunds - Revenue per order and per session +## Architecture (High Level) +``` +CSV (data/) + → DLT ingestion → Snowflake (raw) + → dbt (staging → intermediate → marts) + → KPIs + tests + Elementary report + → Metabase dashboard +``` + +## Modern Data Stack Diagram +![Modern Data Stack](docs/modern data stack.png) + +## Stack +- Ingestion: `dlt` + `polars` +- Transformation: `dbt-core` + `dbt-snowflake` +- Orchestration: `dagster` + `dagster-dbt` +- Data Quality: `dbt tests` + `elementary-data` +- BI: `Metabase` +- Warehouse: `Snowflake` + ## Repository Structure -- `data/` – raw CSVs (sessions, pageviews, orders, items, refunds, products) -- `ingestion/` – DLT pipeline (CSV → Snowflake `raw` schema) -- `dbt_marketing_perf/` – dbt project (staging → intermediate → marts + tests) -- `infra/` – orchestration (Dagster) + Elementary report helper -- `dashboard/` – Metabase dashboard export -- `docs/` – runbooks, optimization notes, KT checklist +- `data/` – raw CSVs + data dictionary (`maven_fuzzy_factory_data_dictionary.csv`) +- `ingestion/` – DLT pipeline (CSV → Snowflake `raw`) +- `dbt_marketing_perf/` – dbt project (staging, intermediate, marts, tests, exposures) +- `infra/` – Dagster orchestration + ops scripts +- `dashboard/` – Metabase dashboard export (PDF) +- `docs/` – user guide, runbook, optimization notes + +## Data Model (dbt) +- **Staging**: source normalization (`stg_*`) +- **Intermediate**: business joins (`int_*`) +- **Marts**: + - **Core**: `fct_sessions`, `fct_orders`, `fct_order_items`, `fct_refunds`, `dim_products`, `dim_users` + - **KPIs**: `kpi_daily_overview`, `kpi_daily_marketing_channels` + +KPI marts are **incremental** with a rolling 3‑day recompute window to keep data fresh. + +## Prerequisites +- Python 3.10+ +- Snowflake access (account, warehouse, database) +- `dbt` and `dagster` installed via `requirements.txt` + +## Configuration +Snowflake environment variables used by dbt (see `dbt_marketing_perf/profiles.yml`): +- `SNOWFLAKE_ACCOUNT` +- `SNOWFLAKE_USER` +- `SNOWFLAKE_PASSWORD` +- `SNOWFLAKE_ROLE` +- `SNOWFLAKE_DATABASE` +- `SNOWFLAKE_WAREHOUSE` + +Notes: +- DLT reads its configuration from `ingestion/.dlt/` (or `infra/.dlt/` when orchestrated). +- `ingestion/filesystem_pipeline.py` uses an absolute path to `data/`. Update `DATA_PATH` if the repo moves. ## Quick Start (Local) - -### 1) Python environment ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt -``` -### 2) Ingest data (DLT) -```bash +# DLT ingestion python ingestion/filesystem_pipeline.py -``` -### 3) dbt build + tests -```bash +# dbt build + tests cd dbt_marketing_perf DBT_PROFILES_DIR=. dbt deps DBT_PROFILES_DIR=. dbt build -``` -### 4) dbt docs -```bash +# dbt docs DBT_PROFILES_DIR=. dbt docs generate DBT_PROFILES_DIR=. dbt docs serve -``` -### 5) Elementary report (local HTML) -```bash +# Elementary report (HTML) ./infra/elementary/run_report.sh ``` -## Orchestration (Dagster) -Event‑driven: **DLT → dbt** via run status sensor. +## Dagster Orchestration +- **Ingestion job**: `dlt_pipeline_job` +- **Transformation job**: `dbt_build_job` (auto-triggered after successful DLT run) +- **Schedule**: daily at 06:00 UTC +Start the Dagster UI: ```bash cd infra -# Start Dagster UI DAGSTER_HOME=$(pwd)/.dagster dagster dev -w dagster_project/workspace.yaml ``` -Jobs: -- `dlt_pipeline_job` -- `dbt_build_job` (triggered automatically on DLT success) - -## dbt Project Highlights -- **Layers**: staging → intermediate → marts (core + kpis) -- **Incremental**: facts + KPI marts -- **Tests**: dbt_utils, dbt_expectations, Elementary schema change monitoring -- **Exposures**: Metabase dashboard + analysis artifacts - -Key models: -- `marts/core/fct_sessions`, `fct_orders`, `fct_order_items`, `fct_refunds` -- `marts/kpis/kpi_daily_marketing_channels`, `kpi_daily_overview` - ## Metabase Dashboard -Public link (local): -- `http://localhost:3000/public/dashboard/998d3922-02d0-44d4-8fcc-4a863cee58a4` - -## CI/CD (GitHub Actions) -- `develop` → `dbt build --target dev` -- `main` → `dbt build --target prod` - -Workflows: -- `.github/workflows/dbt_ci_dev.yml` -- `.github/workflows/dbt_ci_prod.yml` +- PDF export: `dashboard/Metabase - MARKETING PERFORMANCE DASHBOARD.pdf` +- dbt exposure: `marketing_kpis_dashboard` (default local URL in `dbt_marketing_perf/models/exposures.yml`) -## Configuration Notes -- **Snowflake** credentials are read from environment variables (see `dbt_marketing_perf/profiles.yml`). -- **DLT** reads config from `ingestion/.dlt/` (or `infra/.dlt/` if running Dagster locally). -- For production, use secrets manager or CI secrets. Do not hardcode credentials. +## Quality & Monitoring +- Constraints and relationship tests in `schema.yml` +- Freshness/anomaly monitoring via **Elementary** +- Dagster sensors to trigger dbt and alert on failures -## Documentation -- `docs/user_guide.md` – how to run the pipeline -- `docs/ops_optimization.md` – Snowflake optimization guidance -- `docs/kt_checklist.md` – knowledge transfer checklist +## Useful Docs +- `docs/user_guide.md` – daily workflow +- `docs/ops_optimization.md` – Snowflake optimization recommendations +- `docs/kt_checklist.md` – handover checklist -## Suggested Workflow +## Recommended Run Order 1. DLT ingestion -2. dbt build/tests -3. Elementary report -4. Review Metabase KPIs +2. `dbt build` + tests +3. Generate the Elementary report +4. Review the Metabase dashboard --- -If you want this README to include architecture diagrams, SLA/ownership, or deployment steps, tell me and I’ll add them. - -## Architecture (High Level) -``` -CSV files (data/) - | - v - DLT ingestion - | - v -Snowflake (raw schema) - | - v -dbt (staging -> intermediate -> marts) - | - v -KPI tables + exposures - | - +--> Elementary report (HTML) - | - +--> Metabase dashboard -``` - -## SLA and Ownership -- **Ingestion SLA**: daily by 06:00 UTC (Dagster schedule) -- **Transform SLA**: completes within 30 minutes after ingestion -- **Data Quality**: dbt tests + Elementary schema change monitoring -- **Owner**: Analytics Engineering (update owners in `dbt_marketing_perf/models/exposures.yml`) -- **On‑call / escalation**: document in `docs/kt_checklist.md` or your internal runbook - -## Deployment Steps (Prod) - -1) **Provision Snowflake** -- Create warehouse, database, schemas (`raw`, `staging`, `intermediate`, `marts`, `ELEMENTARY`) -- Grant least‑privilege roles to the service account - -2) **Configure Secrets** -- GitHub Actions secrets for Snowflake (`SNOWFLAKE_*`) -- DLT secrets in a secret manager or runtime env (avoid plaintext files) - -3) **Deploy Orchestration (Dagster)** -- Containerize `infra/dagster_project` or deploy on your platform -- Set `DAGSTER_HOME` to persistent storage -- Enable schedule and sensors - -4) **Run Initial Backfill** -- Trigger `dlt_pipeline_job` -- Run `dbt build --target prod` -- Generate Elementary report - -5) **Deploy Metabase** -- Point to `marts` schema -- Configure dashboard refresh after dbt completion - -6) **Enable CI/CD** -- `develop` → dev checks -- `main` → prod dbt build