Update README with project overview and diagram

README.md

@@ -1,159 +1,120 @@
 # Marketing Performance Analysis
 
-End‑to‑end analytics stack for the Maven Fuzzy Factory e‑commerce dataset (teddy bears). This repo ingests CSVs with DLT, transforms with dbt, orchestrates with Dagster, and serves KPIs for a Metabase dashboard.
+End-to-end analytics stack to analyze marketing performance for the **Maven Fuzzy Factory** e-commerce dataset. The project ingests CSVs with **DLT**, transforms with **dbt**, orchestrates with **Dagster**, and surfaces **KPIs** consumable in **Metabase**.
 
-## What This Project Answers
-- Trend of website sessions and orders
-- Session‑to‑order conversion rate over time
-- Best performing marketing channels
+## Business Questions Covered
+- Trends in sessions and orders
+- Session-to-order conversion rate
+- Performance by acquisition channel (utm_source)
+- Revenue, gross margin, and refunds
 - Revenue per order and per session
 
+## Architecture (High Level)
+```
+CSV (data/)
+  → DLT ingestion → Snowflake (raw)
+  → dbt (staging → intermediate → marts)
+  → KPIs + tests + Elementary report
+  → Metabase dashboard
+```
+
+## Modern Data Stack Diagram
+![Modern Data Stack](docs/modern data stack.png)
+
+## Stack
+- Ingestion: `dlt` + `polars`
+- Transformation: `dbt-core` + `dbt-snowflake`
+- Orchestration: `dagster` + `dagster-dbt`
+- Data Quality: `dbt tests` + `elementary-data`
+- BI: `Metabase`
+- Warehouse: `Snowflake`
+
 ## Repository Structure
-- `data/` – raw CSVs (sessions, pageviews, orders, items, refunds, products)
-- `ingestion/` – DLT pipeline (CSV → Snowflake `raw` schema)
-- `dbt_marketing_perf/` – dbt project (staging → intermediate → marts + tests)
-- `infra/` – orchestration (Dagster) + Elementary report helper
-- `dashboard/` – Metabase dashboard export
-- `docs/` – runbooks, optimization notes, KT checklist
+- `data/` – raw CSVs + data dictionary (`maven_fuzzy_factory_data_dictionary.csv`)
+- `ingestion/` – DLT pipeline (CSV → Snowflake `raw`)
+- `dbt_marketing_perf/` – dbt project (staging, intermediate, marts, tests, exposures)
+- `infra/` – Dagster orchestration + ops scripts
+- `dashboard/` – Metabase dashboard export (PDF)
+- `docs/` – user guide, runbook, optimization notes
+
+## Data Model (dbt)
+- **Staging**: source normalization (`stg_*`)
+- **Intermediate**: business joins (`int_*`)
+- **Marts**:
+  - **Core**: `fct_sessions`, `fct_orders`, `fct_order_items`, `fct_refunds`, `dim_products`, `dim_users`
+  - **KPIs**: `kpi_daily_overview`, `kpi_daily_marketing_channels`
+
+KPI marts are **incremental** with a rolling 3‑day recompute window to keep data fresh.
+
+## Prerequisites
+- Python 3.10+
+- Snowflake access (account, warehouse, database)
+- `dbt` and `dagster` installed via `requirements.txt`
+
+## Configuration
+Snowflake environment variables used by dbt (see `dbt_marketing_perf/profiles.yml`):
+- `SNOWFLAKE_ACCOUNT`
+- `SNOWFLAKE_USER`
+- `SNOWFLAKE_PASSWORD`
+- `SNOWFLAKE_ROLE`
+- `SNOWFLAKE_DATABASE`
+- `SNOWFLAKE_WAREHOUSE`
+
+Notes:
+- DLT reads its configuration from `ingestion/.dlt/` (or `infra/.dlt/` when orchestrated).
+- `ingestion/filesystem_pipeline.py` uses an absolute path to `data/`. Update `DATA_PATH` if the repo moves.
 
 ## Quick Start (Local)
-
-### 1) Python environment
 ```bash
 python3 -m venv .venv
 source .venv/bin/activate
 pip install -r requirements.txt
-```
 
-### 2) Ingest data (DLT)
-```bash
+# DLT ingestion
 python ingestion/filesystem_pipeline.py
-```
 
-### 3) dbt build + tests
-```bash
+# dbt build + tests
 cd dbt_marketing_perf
 DBT_PROFILES_DIR=. dbt deps
 DBT_PROFILES_DIR=. dbt build
-```
 
-### 4) dbt docs
-```bash
+# dbt docs
 DBT_PROFILES_DIR=. dbt docs generate
 DBT_PROFILES_DIR=. dbt docs serve
-```
 
-### 5) Elementary report (local HTML)
-```bash
+# Elementary report (HTML)
 ./infra/elementary/run_report.sh
 ```
 
-## Orchestration (Dagster)
-Event‑driven: **DLT → dbt** via run status sensor.
+## Dagster Orchestration
+- **Ingestion job**: `dlt_pipeline_job`
+- **Transformation job**: `dbt_build_job` (auto-triggered after successful DLT run)
+- **Schedule**: daily at 06:00 UTC
 
+Start the Dagster UI:
 ```bash
 cd infra
-# Start Dagster UI
 DAGSTER_HOME=$(pwd)/.dagster dagster dev -w dagster_project/workspace.yaml
 ```
 
-Jobs:
-- `dlt_pipeline_job`
-- `dbt_build_job` (triggered automatically on DLT success)
-
-## dbt Project Highlights
-- **Layers**: staging → intermediate → marts (core + kpis)
-- **Incremental**: facts + KPI marts
-- **Tests**: dbt_utils, dbt_expectations, Elementary schema change monitoring
-- **Exposures**: Metabase dashboard + analysis artifacts
-
-Key models:
-- `marts/core/fct_sessions`, `fct_orders`, `fct_order_items`, `fct_refunds`
-- `marts/kpis/kpi_daily_marketing_channels`, `kpi_daily_overview`
-
 ## Metabase Dashboard
-Public link (local):
-- `http://localhost:3000/public/dashboard/998d3922-02d0-44d4-8fcc-4a863cee58a4`
-
-## CI/CD (GitHub Actions)
-- `develop` → `dbt build --target dev`
-- `main` → `dbt build --target prod`
-
-Workflows:
-- `.github/workflows/dbt_ci_dev.yml`
-- `.github/workflows/dbt_ci_prod.yml`
+- PDF export: `dashboard/Metabase - MARKETING PERFORMANCE DASHBOARD.pdf`
+- dbt exposure: `marketing_kpis_dashboard` (default local URL in `dbt_marketing_perf/models/exposures.yml`)
 
-## Configuration Notes
-- **Snowflake** credentials are read from environment variables (see `dbt_marketing_perf/profiles.yml`).
-- **DLT** reads config from `ingestion/.dlt/` (or `infra/.dlt/` if running Dagster locally).
-- For production, use secrets manager or CI secrets. Do not hardcode credentials.
+## Quality & Monitoring
+- Constraints and relationship tests in `schema.yml`
+- Freshness/anomaly monitoring via **Elementary**
+- Dagster sensors to trigger dbt and alert on failures
 
-## Documentation
-- `docs/user_guide.md` – how to run the pipeline
-- `docs/ops_optimization.md` – Snowflake optimization guidance
-- `docs/kt_checklist.md` – knowledge transfer checklist
+## Useful Docs
+- `docs/user_guide.md` – daily workflow
+- `docs/ops_optimization.md` – Snowflake optimization recommendations
+- `docs/kt_checklist.md` – handover checklist
 
-## Suggested Workflow
+## Recommended Run Order
 1. DLT ingestion
-2. dbt build/tests
-3. Elementary report
-4. Review Metabase KPIs
+2. `dbt build` + tests
+3. Generate the Elementary report
+4. Review the Metabase dashboard
 
 ---
-If you want this README to include architecture diagrams, SLA/ownership, or deployment steps, tell me and I’ll add them.
-
-## Architecture (High Level)
-```
-CSV files (data/)
-        |
-        v
-     DLT ingestion
-        |
-        v
-Snowflake (raw schema)
-        |
-        v
-dbt (staging -> intermediate -> marts)
-        |
-        v
-KPI tables + exposures
-        |
-        +--> Elementary report (HTML)
-        |
-        +--> Metabase dashboard
-```
-
-## SLA and Ownership
-- **Ingestion SLA**: daily by 06:00 UTC (Dagster schedule)
-- **Transform SLA**: completes within 30 minutes after ingestion
-- **Data Quality**: dbt tests + Elementary schema change monitoring
-- **Owner**: Analytics Engineering (update owners in `dbt_marketing_perf/models/exposures.yml`)
-- **On‑call / escalation**: document in `docs/kt_checklist.md` or your internal runbook
-
-## Deployment Steps (Prod)
-
-1) **Provision Snowflake**
-- Create warehouse, database, schemas (`raw`, `staging`, `intermediate`, `marts`, `ELEMENTARY`)
-- Grant least‑privilege roles to the service account
-
-2) **Configure Secrets**
-- GitHub Actions secrets for Snowflake (`SNOWFLAKE_*`)
-- DLT secrets in a secret manager or runtime env (avoid plaintext files)
-
-3) **Deploy Orchestration (Dagster)**
-- Containerize `infra/dagster_project` or deploy on your platform
-- Set `DAGSTER_HOME` to persistent storage
-- Enable schedule and sensors
-
-4) **Run Initial Backfill**
-- Trigger `dlt_pipeline_job`
-- Run `dbt build --target prod`
-- Generate Elementary report
-
-5) **Deploy Metabase**
-- Point to `marts` schema
-- Configure dashboard refresh after dbt completion
-
-6) **Enable CI/CD**
-- `develop` → dev checks
-- `main` → prod dbt build