diff --git a/CHANGELOG.md b/CHANGELOG.md index b4ee7a8..c32302d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ All notable changes to PipelineProbe are documented here. -The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). +Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Versioning follows [Semantic Versioning](https://semver.org/). --- @@ -10,6 +10,10 @@ Versioning follows [Semantic Versioning](https://semver.org/). ## [Unreleased] ### Added +- **Report UI overhaul** — sticky topbar, animated SVG health-score ring, severity filter buttons (All / Critical / Warnings / Info), affected-resource tags on each finding card, subtle per-severity background tints, and responsive print styles. +- **Info count card** — the summary grid now shows a fourth card for informational findings alongside Critical and Warnings. +- `info_count` included in the JSON report summary for downstream tooling. +- `ring_offset` computed in the renderer and injected into the Jinja2 template so the SVG ring accurately reflects the health score. - BigQuery connector querying `INFORMATION_SCHEMA.TABLE_STORAGE` and `COLUMNS` for real `has_timestamps` detection. - Snowflake connector using a CTE pattern to work around correlated subquery restrictions. - `python-dateutil` declared as an explicit dependency in `pyproject.toml`. @@ -34,12 +38,12 @@ Versioning follows [Semantic Versioning](https://semver.org/). ## [0.1.0] — 2026-03-15 ### Added -- Phase 0: Project skeleton, `pyproject.toml`, CI workflow (ruff + pytest), CLI entry point. -- Phase 1 (MVP connectors): +- **Phase 0** — Project skeleton, `pyproject.toml`, CI workflow (ruff + pytest), CLI entry point. +- **Phase 1 — MVP connectors:** - `AirflowConnector` — fetches DAGs, DAG runs, and task configurations via the Airflow REST API. - `DbtConnector` — reads `manifest.json` and `run_results.json`; counts tests per model. - `PostgresConnector` — queries `pg_stat_user_tables` and `information_schema.columns`. -- Phase 1 (Rules Engine): +- **Phase 1 — Rules Engine:** - `check_missing_retries` — warns on tasks with no retry configuration. - `check_missing_slas` — informs on tasks with no SLA. - `check_high_failure_rate` — critical alert for DAGs with >20% failure rate over ≥5 runs. @@ -48,13 +52,20 @@ Versioning follows [Semantic Versioning](https://semver.org/). - `check_failing_models` — critical alert for dbt models that failed their last run. - `check_large_tables` — warns on tables with >10M rows. - `check_missing_timestamps` — warns on tables >1M rows without `created_at`/`updated_at`. -- Phase 1 (Report Renderer): Jinja2 HTML template; JSON output via `model_dump()`. -- Phase 2 (DX & CI): +- **Phase 1 — Report Renderer** — Jinja2 HTML template; JSON output via `model_dump()`. +- **Phase 2 — DX & CI:** - `pipelineprobe init` command generating a default `pipelineprobe.yml`. - `--fail-on-critical` and `--format` CLI overrides. - - `examples/github_action.yml` GitHub Actions workflow. + - `examples/github-actions/pipelineprobe.yml` GitHub Actions workflow. - `docs/ci-integration.md`. - `.gitignore` with standard Python exclusions. -- Phase 3 (Extended Integrations): +- **Phase 3 — Extended Integrations:** - `BigQueryConnector`. - `SnowflakeConnector`. + - `pipelineprobe doctor` command for connectivity validation. + - `pipelineprobe diff` command for regression detection between two JSON reports. + +--- + +[Unreleased]: https://github.com/willowvibe/pipelineprobe/compare/v0.1.0...HEAD +[0.1.0]: https://github.com/willowvibe/pipelineprobe/releases/tag/v0.1.0 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 502b1ee..e71677f 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ # Contributing to PipelineProbe -Thank you for your interest in contributing! PipelineProbe is an open-source project by [WillowVibe](https://www.willowvibe.com) and we welcome contributions of all kinds, including bug reports, documentation improvements, and new connectors. +Thank you for your interest in contributing! PipelineProbe is an open-source project by [WillowVibe](https://www.willowvibe.com) and we welcome contributions of all kinds — bug reports, documentation improvements, new rules, and new connectors. --- @@ -18,12 +18,10 @@ Thank you for your interest in contributing! PipelineProbe is an open-source pro git clone https://github.com/willowvibe/pipelineprobe.git cd pipelineprobe -# Create a virtual environment python -m venv .venv source .venv/bin/activate # Linux / macOS # .venv\Scripts\activate # Windows -# Install in editable mode with all dev dependencies pip install -e ".[dev]" ``` @@ -35,21 +33,28 @@ pip install -e ".[dev]" pytest ``` -Tests live in the `tests/` directory and use `pytest`. Fixtures for mock data are in `tests/conftest.py`. +Tests live in `tests/` and use `pytest`. Fixtures for mock data are in `tests/conftest.py`. + +To run a specific file or test: + +```bash +pytest tests/test_rules.py +pytest tests/test_renderer.py::test_render_html_no_issues +``` --- -## Linting +## Linting & Formatting -We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting: +We use [ruff](https://docs.astral.sh/ruff/) for both linting and formatting: ```bash -ruff check . # Check for lint errors -ruff check --fix . # Auto-fix safe issues -ruff format . # Format code +ruff check . # report lint errors +ruff check --fix . # auto-fix safe issues +ruff format . # format all files ``` -CI will fail if `ruff check .` reports any errors. +CI fails if `ruff check .` reports any errors. Run both before opening a PR. --- @@ -58,24 +63,30 @@ CI will fail if `ruff check .` reports any errors. ``` pipelineprobe/ ├── pipelineprobe/ -│ ├── cli.py # CLI entry points (audit, init) -│ ├── config.py # YAML + env var config loader (Pydantic) -│ ├── models.py # Core domain models (Dag, Task, DbtModel, Issue) -│ ├── renderer.py # HTML + JSON report renderer (Jinja2) +│ ├── cli.py # CLI entry points (audit, init, doctor, diff) +│ ├── config.py # YAML + env var config loader (Pydantic) +│ ├── models.py # Core domain models (Dag, Task, DbtModel, Issue) +│ ├── renderer.py # HTML + JSON report renderer (Jinja2) │ ├── connectors/ -│ │ ├── airflow.py # Apache Airflow REST API connector -│ │ ├── dbt.py # dbt manifest / run_results reader -│ │ ├── postgres.py # PostgreSQL INFORMATION_SCHEMA queries -│ │ ├── bigquery.py # BigQuery INFORMATION_SCHEMA queries -│ │ └── snowflake.py # Snowflake INFORMATION_SCHEMA queries +│ │ ├── airflow.py # Apache Airflow REST API connector +│ │ ├── dbt.py # dbt manifest / run_results reader +│ │ ├── postgres.py # PostgreSQL INFORMATION_SCHEMA queries +│ │ ├── bigquery.py # BigQuery INFORMATION_SCHEMA queries +│ │ └── snowflake.py # Snowflake INFORMATION_SCHEMA queries │ ├── rules/ -│ │ ├── engine.py # Pluggable rule registration and execution +│ │ ├── engine.py # Pluggable rule registration and execution │ │ ├── airflow_rules.py │ │ ├── dbt_rules.py │ │ └── postgres_rules.py │ └── templates/ -│ └── report.html # Jinja2 HTML report template +│ └── report.html # Jinja2 HTML report template ├── tests/ +│ ├── conftest.py # Shared pytest fixtures +│ ├── test_cli.py +│ ├── test_config.py +│ ├── test_connectors.py +│ ├── test_renderer.py +│ └── test_rules.py ├── docs/ ├── examples/ └── pyproject.toml @@ -88,18 +99,20 @@ pipelineprobe/ 1. Add a function to the appropriate `rules/` module (e.g. `airflow_rules.py`): ```python +from typing import List +from pipelineprobe.models import Issue + def check_my_new_rule(context: dict) -> List[Issue]: issues = [] for dag in context.get("airflow_dags", []): - # Your logic here - if some_condition: + if some_condition(dag): issues.append(Issue( severity="warning", category="dag", - summary="...", - details="...", - recommendation="...", - affected_resources=[dag.id] + summary="Short description of the problem", + details="Longer explanation with context.", + recommendation="Concrete steps to fix it.", + affected_resources=[dag.id], )) return issues ``` @@ -112,39 +125,62 @@ def register_airflow_rules(engine): engine.register_rule(check_my_new_rule) ``` -3. Write a test in `tests/test_rules.py`. +3. Write tests in `tests/test_rules.py` — at minimum one happy-path and one edge-case test. + +4. If the rule is configurable (e.g. severity override), document it in `docs/configuration.md` under the `rules.severity_overrides` table. --- ## Adding a New Connector 1. Create `pipelineprobe/connectors/my_connector.py`. -2. Implement a class with a `get_stats_sync() -> List[Dict[str, Any]]` method that returns rows with `schemaname`, `tablename`, `row_count`, and `has_timestamps`. -3. Add your new warehouse type to `cli.py` in the `if cfg.warehouse.type ==` block. +2. Implement a class with a `get_stats_sync() -> List[Dict[str, Any]]` method returning rows with at least: `schemaname`, `tablename`, `row_count`, `has_timestamps`. +3. Route `cfg.warehouse.type == "my_connector"` in the `audit()` function in `cli.py`. 4. Add any required credentials to `WarehouseConfig` in `config.py`. -5. Document it in `docs/configuration.md`. +5. Document the new connector and its config fields in `docs/configuration.md`. +6. Add at least one connectivity test in `tests/test_connectors.py`. + +--- + +## Working on the HTML Report + +The report template lives at `pipelineprobe/templates/report.html` and is rendered by `renderer.py` using Jinja2. + +Variables available in the template: + +| Variable | Type | Description | +|---|---|---| +| `issues` | `List[Issue]` | All findings from the rule engine | +| `summary` | `dict` | Health score, counts, dag_count, metadata | +| `top_actions` | `List[Issue]` | Top 3 critical/warning issues | +| `metadata` | `dict` | orchestrator_url, warehouse_type, dbt_target | +| `generated_at` | `str` | Formatted timestamp string | +| `version` | `str` | PipelineProbe version | +| `ring_offset` | `float` | SVG ring stroke-dashoffset (0 = full, 251.3 = empty) | + +To preview a rendered report, run a real or mock audit and open `./reports/pipelineprobe-report.html` in a browser. --- ## Pull Request Process 1. Fork the repo and create your branch from `main`. -2. Write tests for your changes (aim for at least one happy-path and one edge-case test). +2. Write tests for your changes. 3. Ensure `ruff check .` and `pytest` both pass. -4. Open a Pull Request with a clear description of: +4. Open a PR with a description covering: - **What** changed - - **Why** it's needed + - **Why** it is needed - **How** to test it manually --- ## Reporting Bugs -Please open a GitHub Issue with: +Open a GitHub Issue with: - PipelineProbe version (`pip show pipelineprobe`) - Python version -- A description of the bug and what you expected to happen -- Sanitized config (no credentials) and any relevant log output +- A clear description of what happened vs. what you expected +- Sanitized config (no credentials) and relevant log output --- diff --git a/README.md b/README.md index 15e4175..5439f0e 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,30 @@ # PipelineProbe -> *Instant Data Pipeline Audit Report for Airflow + dbt + modern warehouses* +> *Instant Data Pipeline Audit — Airflow · dbt · modern warehouses* [![CI](https://github.com/willowvibe/pipelineprobe/actions/workflows/ci.yml/badge.svg)](https://github.com/willowvibe/pipelineprobe/actions/workflows/ci.yml) +[![PyPI](https://img.shields.io/pypi/v/pipelineprobe.svg)](https://pypi.org/project/pipelineprobe/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/) -PipelineProbe is a **read-only, one-command audit tool** for data pipelines. It connects to your existing stack — Apache Airflow, dbt, and a modern warehouse — and produces a single actionable HTML or JSON report surfacing critical issues, missing SLAs, missing tests, high failure rates, and more. +PipelineProbe is a **read-only, one-command audit tool** for data pipelines. It connects to your existing stack — Apache Airflow, dbt, and a modern warehouse — and produces a single actionable HTML or JSON report surfacing critical issues, missing SLAs, failing tests, high failure rates, and more. -It is open-sourced by [WillowVibe](https://www.willowvibe.com). +Open-sourced by [WillowVibe](https://www.willowvibe.com). --- -## ✨ Features +## Features | Area | What it checks | |---|---| | **Airflow** | DAG failure rates, missing retries, missing SLAs, stale pipelines, alert configuration | -| **dbt** | Models with zero tests, test failure ratio, failing last runs | -| **Warehouse** | Largest tables, tables missing audit timestamps (`created_at`/`updated_at`) | -| **Report** | Health score (0–100), critical / warning / info counts, HTML + JSON output | +| **dbt** | Models with zero tests, failing last runs | +| **Warehouse** | Largest tables, tables missing audit timestamps (`created_at` / `updated_at`) | +| **Report** | Health score (0–100), critical / warning / info counts, severity filter, HTML + JSON output | --- -## 🚀 Quick Start +## Quick Start ### 1. Install @@ -31,13 +32,13 @@ It is open-sourced by [WillowVibe](https://www.willowvibe.com). pip install pipelineprobe ``` -### 2. Initialize a config file +### 2. Create a config file ```bash pipelineprobe init ``` -This creates `pipelineprobe.yml` in the current directory. +This writes `pipelineprobe.yml` to the current directory with sensible defaults. ### 3. Run an audit @@ -45,21 +46,24 @@ This creates `pipelineprobe.yml` in the current directory. pipelineprobe audit --config pipelineprobe.yml ``` -Reports are written to `./reports/` by default. +Reports are written to `./reports/` by default — open `pipelineprobe-report.html` in any browser. -### ⏱️ 5-Minute Quickstart +### 5-Minute Docker Quickstart + +Want to see it in action without a local stack? -Want to see it in action without a local stack? Try our [Quickstart Example](examples/quickstart/README.md): ```bash cd examples/quickstart docker compose up --build ``` +See [examples/quickstart/README.md](examples/quickstart/README.md) for details. + --- -## ⚙️ Configuration +## Configuration -See [docs/configuration.md](docs/configuration.md) for the full reference. A minimal example: +A minimal `pipelineprobe.yml`: ```yaml orchestrator: @@ -75,8 +79,6 @@ dbt: warehouse: type: postgres # postgres | bigquery | snowflake dsn: "postgresql://user:pass@localhost:5432/analytics" - # for BigQuery: project_id: "my-gcp-project" - # for Snowflake: account: "xyz.us-east-1", username: "...", password: "..." report: output_dir: "./reports" @@ -84,98 +86,149 @@ report: fail_on_critical: 5 ``` -### CLI Flags +See [docs/configuration.md](docs/configuration.md) for the full reference including BigQuery, Snowflake, and all rule-level options. -| Flag | Description | -|---|---| -| `--config` | Path to config YAML (default: `pipelineprobe.yml`) | -| `--format` | Override output format: `html`, `json`, or `both` | -| `--fail-on-critical` | Override the critical issue threshold for CI exits | -| `--version` | Show version and exit | +--- -### CLI Commands +## CLI Reference + +### Commands | Command | Description | |---|---| -| `init` | Initialize a default `pipelineprobe.yml` | -| `audit` | Run the full audit pipeline | -| `doctor` | Validate connectivity to source systems | +| `pipelineprobe init` | Write a default `pipelineprobe.yml` to the current directory | +| `pipelineprobe audit` | Run the full audit and generate reports | +| `pipelineprobe doctor` | Validate connectivity to Airflow, dbt, and the warehouse before auditing | +| `pipelineprobe diff ` | Compare two JSON reports and surface regressions / improvements | + +### `audit` flags + +| Flag | Default | Description | +|---|---|---| +| `--config FILE` | `pipelineprobe.yml` | Path to config YAML | +| `--format FORMAT` | from config | Override output format: `html`, `json`, or `both` | +| `--fail-on-critical N` | from config | Override critical issue threshold for non-zero exit | +| `--version` | — | Print version and exit | + +### Exit codes + +| Code | Meaning | +|---|---| +| `0` | Audit completed; critical count at or below threshold | +| `1` | Critical issue count exceeds `fail_on_critical`, or a config / connectivity error occurred | + +### Examples + +```bash +# Basic local audit +pipelineprobe audit + +# CI strict mode: fail if even one critical issue is found +pipelineprobe audit --format both --fail-on-critical 0 + +# Compare today's report against yesterday's baseline +pipelineprobe diff reports/baseline.json reports/report.json + +# Check connectivity without running the full audit +pipelineprobe doctor --config staging.yml + +# Print version +pipelineprobe --version +``` --- -## 🔌 Supported Integrations +## Supported Integrations | Connector | Status | |---|---| -| Apache Airflow (REST API ≥ 2.0) | ✅ Supported | -| dbt Core (manifest + run_results) | ✅ Supported | -| PostgreSQL | ✅ Supported | -| BigQuery | ✅ Supported | -| Snowflake | ✅ Supported | +| Apache Airflow (REST API ≥ 2.0) | Supported | +| dbt Core (manifest + run_results) | Supported | +| PostgreSQL | Supported | +| BigQuery | Supported | +| Snowflake | Supported | --- -## 🔄 Standard Workflows +## Standard Workflows + +### 1. Local Audit + +Identify issues before they reach production. Run `pipelineprobe audit` on a dev machine or before merging infrastructure changes. -### 1. Local Audit (Internal Teams) -Identify issues before they hit production. Run `pipelineprobe audit` locally or manually on a dev machine to verify current infra state. +```bash +pipelineprobe init +# edit pipelineprobe.yml with your Airflow URL, dbt paths, and warehouse DSN +pipelineprobe audit --format html +# open ./reports/pipelineprobe-report.html +``` ### 2. CI Quality Gate -Fail your build when critical issues surface. Use the `--fail-on-critical 0` flag to enforce strict standards. See [CI Guide](docs/ci-integration.md). -### 3. Consulting / One-off Audits -Perfect for external auditors or consultants. Connect to a client's Airflow/Postgres once, run the audit, and provide the polished HTML report as a deliverable. +Fail your build when critical issues surface. Use `--fail-on-critical 0` for zero-tolerance enforcement. See [docs/ci-integration.md](docs/ci-integration.md). ---- +```bash +pipelineprobe audit --format both --fail-on-critical 0 +``` -## 🆚 Comparison +### 3. Regression Detection with `diff` -How is PipelineProbe different from full observability platforms? +Compare successive audit reports to catch new issues introduced between runs: -| Feature | Monitoring Tools (Datadog, Monte Carlo) | Quality Libraries (Soda, GE) | **PipelineProbe** | -|---|---|---|---| -| **Focus** | Continuous monitoring & alerting | Row-level data validation | Infrastructure & config audit | -| **Effort** | High (setup agents/SDKs) | Medium (write YAML expectations) | **Zero (read-only API/metastore)** | -| **Best For** | On-call engineers | Data engineers | **Consultants / Team Leads** | +```bash +pipelineprobe audit --format json # generates report.json +# ... later / next CI run ... +pipelineprobe diff reports/baseline.json reports/report.json +# exits 1 if regressions found +``` + +### 4. Consulting / One-off Audits + +Connect to a client's stack, run the audit, and deliver the polished HTML report as a professional-grade artefact. --- -## 🤖 CI/CD Integration +## How PipelineProbe Differs from Other Tools -PipelineProbe can automatically fail your CI pipeline when critical issues exceed your threshold. See [docs/ci-integration.md](docs/ci-integration.md) for GitHub Actions and GitLab CI examples. +| Feature | Monitoring (Datadog, Monte Carlo) | Quality Libraries (Soda, Great Expectations) | **PipelineProbe** | +|---|---|---|---| +| **Focus** | Continuous alerting & dashboards | Row-level data validation | Infrastructure & config audit | +| **Effort to start** | High — install agents, configure SDKs | Medium — write YAML expectations | **Zero — read-only, no agents** | +| **Output** | Dashboards, alerts | Pass / fail per expectation | **Single HTML / JSON report** | +| **Best for** | On-call engineers | Data engineers | **Consultants · Team leads · CI gates** | --- -## 🖼️ Report Preview +## CI/CD Integration -![PipelineProbe HTML Report Screenshot](docs/report-screenshot.png) +PipelineProbe can automatically fail your CI pipeline when critical issues exceed your threshold. See [docs/ci-integration.md](docs/ci-integration.md) for ready-to-use GitHub Actions and GitLab CI configs. --- -## 📖 Documentation +## Documentation | Document | Description | |---|---| -| [Configuration Reference](docs/configuration.md) | All YAML and environment variable options | -| [CI Integration Guide](docs/ci-integration.md) | GitHub Actions, GitLab CI, fail-on-critical | +| [Configuration Reference](docs/configuration.md) | All YAML fields, environment variables, and rule-level options | +| [CI Integration Guide](docs/ci-integration.md) | GitHub Actions, GitLab CI, `diff` regression detection | | [Architecture](docs/architecture.md) | How connectors, rules, and the renderer fit together | -| [Contributing](CONTRIBUTING.md) | Development setup, testing, PRs | +| [Contributing](CONTRIBUTING.md) | Development setup, testing, adding rules and connectors | | [Changelog](CHANGELOG.md) | Release history | --- -## 🗺️ Roadmap +## Roadmap -- [ ] **v0.2.0**: Prefect and Dagster connectors. -- [ ] **v0.3.0**: Basic cost insights (scanned bytes for BQ/Snowflake). -- [ ] **v1.0.0**: Comprehensive data lineage support. +- [ ] **v0.2.0** — Prefect and Dagster connectors +- [ ] **v0.3.0** — Cost insights (scanned bytes for BigQuery / Snowflake) +- [ ] **v1.0.0** — Data lineage support --- -## 🤝 Contributing +## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) for how to get started. +See [CONTRIBUTING.md](CONTRIBUTING.md) to get started. All contributions are welcome — bug reports, docs, new rules, new connectors. -## 📄 License +## License -MIT License — see [LICENSE](LICENSE) for details. +MIT — see [LICENSE](LICENSE) for details. diff --git a/docs/architecture.md b/docs/architecture.md index e64f404..4afdef7 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,48 +1,43 @@ # Architecture -This document explains how PipelineProbe's components fit together. +This document explains how PipelineProbe's components fit together and how to extend them. --- ## High-Level Overview ``` - +-----------+ - pipelineprobe | CLI | audit / init - +-----------+ - | - loads | PipelineProbeConfig - v - +------------------------------------+ - | Connectors Layer | - | AirflowConnector (REST API) | - | DbtConnector (JSON files) | - | PostgresConnector (asyncpg) | - | BigQueryConnector (google-cloud) | - | SnowflakeConnector(sf.connector) | - +------------------------------------+ - | - returns | Dag, Task, DbtModel, TableStat dicts - v - +------------------------------------+ - | Rules Engine | - | airflow_rules dbt_rules | - | postgres_rules ... | - +------------------------------------+ - | - emits | List[Issue] - v - +------------------------------------+ - | Report Renderer | - | Jinja2 HTML template | - | json.dump report.json | - +------------------------------------+ - | - writes | - v - ./reports/ - pipelineprobe-report.html - report.json +┌──────────────────────────────────────────────────┐ +│ CLI │ +│ audit · init · doctor · diff │ +└────────────────────────┬─────────────────────────┘ + │ PipelineProbeConfig + ▼ +┌──────────────────────────────────────────────────┐ +│ Connectors Layer │ +│ AirflowConnector (Airflow REST API ≥ 2.0) │ +│ DbtConnector (manifest.json / run_results)│ +│ PostgresConnector (pg_stat_user_tables) │ +│ BigQueryConnector (INFORMATION_SCHEMA) │ +│ SnowflakeConnector (INFORMATION_SCHEMA + CTE) │ +└────────────────────────┬─────────────────────────┘ + │ Dag, Task, DbtModel, TableStat dicts + ▼ +┌──────────────────────────────────────────────────┐ +│ Rules Engine │ +│ airflow_rules dbt_rules postgres_rules │ +└────────────────────────┬─────────────────────────┘ + │ List[Issue] + ▼ +┌──────────────────────────────────────────────────┐ +│ Report Renderer │ +│ Jinja2 HTML template · json.dump │ +└────────────────────────┬─────────────────────────┘ + │ + ▼ + ./reports/ + pipelineprobe-report.html + report.json ``` --- @@ -54,43 +49,55 @@ This document explains how PipelineProbe's components fit together. Entry point registered as `pipelineprobe` via `pyproject.toml`. Uses [Typer](https://typer.tiangolo.com/). **Commands:** -- `pipelineprobe init` — writes a default `pipelineprobe.yml` to the current directory. -- `pipelineprobe audit` — runs the full pipeline: load config → collect data → run rules → render reports → exit with code 1 if critical threshold exceeded. + +| Command | Description | +|---|---| +| `pipelineprobe init` | Write a default `pipelineprobe.yml` to the current directory | +| `pipelineprobe audit` | Full pipeline: load config → collect → run rules → render → exit with code 1 if threshold exceeded | +| `pipelineprobe doctor` | Validate connectivity to Airflow, dbt artifacts, and the warehouse without running a full audit | +| `pipelineprobe diff ` | Compare two JSON reports; exit 1 if regressions found | ### Config (`config.py`) -Pydantic models backed by `pydantic-settings`. YAML values are loaded first; environment variables take precedence. +Pydantic models backed by `pydantic-settings`. YAML values load first; environment variables override. Key models: -- `AirflowConfig` -- `DbtConfig` -- `WarehouseConfig` — type-dispatched at runtime in `cli.py` -- `ReportConfig` + +| Model | Purpose | +|---|---| +| `AirflowConfig` | Airflow REST API URL, credentials, SSL, lookback window | +| `DbtConfig` | dbt project directory, manifest / run_results paths | +| `WarehouseConfig` | Warehouse type + credentials (dispatched at runtime) | +| `ReportConfig` | Output directory, format, fail_on_critical threshold | +| `RulesConfig` | Staleness threshold, fetch concurrency, per-rule severity overrides | ### Connectors (`connectors/`) -Each connector is responsible for fetching raw data from one source system and returning Python dicts or domain model objects. +Each connector fetches raw data from one source and returns Python objects or dicts. All connectors catch exceptions internally and return an empty list on failure, ensuring a **partial report** is still produced when one source system is unavailable. | Connector | Input | Output | |---|---|---| -| `AirflowConnector` | Airflow REST API | `List[Dag]`, `List[Task]` | +| `AirflowConnector` | Airflow REST API (paginated) | `List[Dag]`, `List[Task]` | | `DbtConnector` | `manifest.json`, `run_results.json` | `List[DbtModel]` | | `PostgresConnector` | `pg_stat_user_tables` + `information_schema` | `List[Dict]` | | `BigQueryConnector` | `INFORMATION_SCHEMA.TABLE_STORAGE` + `COLUMNS` | `List[Dict]` | | `SnowflakeConnector` | `INFORMATION_SCHEMA.TABLES` + CTE over `COLUMNS` | `List[Dict]` | -All connectors catch exceptions internally and return an empty list on failure, ensuring a **partial report** is still produced when one source system is unavailable. +The Airflow connector uses `asyncio` to fetch DAG run history and task configs concurrently (controlled by `rules.fetch_concurrency`). ### Rules Engine (`rules/engine.py`, `rules/*_rules.py`) -The `RuleEngine` is a simple registry of callable rule functions. Each rule receives a `context` dict and returns `List[Issue]`. +`RuleEngine` is a simple registry of callable rule functions. Each rule receives a `context` dict and returns `List[Issue]`. ```python context = { - "airflow_dags": [Dag, ...], - "airflow_tasks": [Task, ...], - "dbt_models": [DbtModel, ...], - "postgres_tables": [{"tablename": ..., "row_count": ..., ...}, ...], + "airflow_dags": [Dag, ...], + "airflow_tasks": [Task, ...], + "dbt_models": [DbtModel, ...], + "warehouse_tables": [{"tablename": ..., "row_count": ..., ...}, ...], + "warehouse_type": "postgres", + "rule_severity_overrides": {...}, + "stale_threshold_days": 7, } ``` @@ -98,9 +105,9 @@ Rules are registered at module import time via `register_*_rules(engine)`. New r ### Domain Models (`models.py`) -Pydantic models used throughout: +Pydantic models used throughout the pipeline: -| Model | Fields | +| Model | Key Fields | |---|---| | `DagRun` | `state`, `start_time`, `end_time` | | `Dag` | `id`, `is_active`, `recent_runs`, `owner` | @@ -110,9 +117,15 @@ Pydantic models used throughout: ### Report Renderer (`renderer.py`) -`ReportRenderer` accepts `List[Issue]` + a summary dict and produces: -- `pipelineprobe-report.html` via Jinja2 (`templates/report.html`) -- `report.json` via `json.dump` with a custom serializer for non-standard types (e.g. `timedelta`) +`ReportRenderer` accepts `List[Issue]` and a summary dict and produces: + +- **`pipelineprobe-report.html`** — rendered via Jinja2 from `templates/report.html` +- **`report.json`** — serialized with `json.dump` and a custom `timedelta` handler + +Before rendering HTML, the renderer computes: + +- `top_actions` — the top 3 critical/warning issues sorted by severity then category +- `ring_offset` — SVG stroke-dashoffset for the health score ring (circumference 251.3 × (1 − score/100)) --- @@ -121,28 +134,43 @@ Pydantic models used throughout: ``` audit() │ - ├─ load_config() → PipelineProbeConfig + ├─ load_config() → PipelineProbeConfig │ ├─ AirflowConnector - │ ├─ get_dags() → List[Dag] (paginated) - │ ├─ get_dag_runs(id) → Dag.recent_runs populated - │ └─ get_tasks(id) → List[Task] + │ ├─ get_dags() → List[Dag] (paginated, all pages) + │ └─ fetch_dag_details() → Dag.recent_runs + List[Task] (concurrent) │ ├─ DbtConnector - │ └─ get_models() → List[DbtModel] + │ └─ get_models() → List[DbtModel] │ ├─ WarehouseConnector - │ └─ get_stats_sync() → List[Dict] + │ └─ get_stats_sync() → List[Dict] │ - ├─ RuleEngine.run(context)→ List[Issue] + ├─ RuleEngine.run(context) → List[Issue] │ - ├─ compute summary (score, counts) + ├─ compute summary + │ score, critical_count, warning_count, info_count, + │ total_issues, dag_count, score_formula, metadata │ └─ ReportRenderer - ├─ render_html() → ./reports/pipelineprobe-report.html - └─ render_json() → ./reports/report.json + ├─ render_html() → ./reports/pipelineprobe-report.html + └─ render_json() → ./reports/report.json +``` + +### Health Score Formula + +``` +critical_density = critical_count / dag_count +warning_density = warning_count / dag_count + +critical_penalty = min(90, critical_density × 200) +warning_penalty = min(20, warning_density × 40) + +score = max(0, round(100 − critical_penalty − warning_penalty)) ``` +Normalising by `dag_count` means five criticals in a 500-DAG platform is a very different signal from five criticals in a 10-DAG shop. + --- ## Extending PipelineProbe @@ -151,11 +179,16 @@ audit() 1. Create `pipelineprobe/connectors/my_connector.py`. 2. Implement `get_stats_sync() -> List[Dict[str, Any]]` returning rows with `schemaname`, `tablename`, `row_count`, `has_timestamps`. -3. Route `cfg.warehouse.type == "my_connector"` in `cli.py`. +3. Route `cfg.warehouse.type == "my_connector"` in the `audit()` function in `cli.py`. 4. Add credentials to `WarehouseConfig` in `config.py`. +5. Document in `docs/configuration.md`. ### Adding a Rule 1. Write `def check_*(context: dict) -> List[Issue]` in the appropriate `rules/` file. 2. Call `engine.register_rule(check_*)` in the `register_*_rules` function. 3. Add a test in `tests/test_rules.py`. + +### Modifying the HTML Report + +The report template is `pipelineprobe/templates/report.html`. Template variables are documented in [CONTRIBUTING.md](../CONTRIBUTING.md#working-on-the-html-report). After editing, render a fresh report with `pipelineprobe audit` and inspect it in a browser. diff --git a/docs/bugs_improvements.md b/docs/bugs_improvements.md index 4fbeeeb..be49ad5 100644 --- a/docs/bugs_improvements.md +++ b/docs/bugs_improvements.md @@ -1,81 +1,65 @@ -Right now PipelineProbe is structurally solid and safe to run; the next step is to make it “drop‑in usable” with great UX and examples rather than more core code changes. - -## 1. Make setup truly plug‑and‑play - -1. Add a minimal “quickstart” example project: - - Tiny docker‑compose with: Airflow + Postgres + a toy dbt project + PipelineProbe container. - - One command: `docker compose up` and a README section: “Run your first audit in 5 minutes”. - - This is what converts visitors into actual users; all successful CLI tools do this. [dev](https://dev.to/wesen/14-great-tips-to-make-amazing-cli-applications-3gp3) - -2. Harden config UX: - - Document all CLI flags and YAML fields in README (table: field, type, default, env override). - - In `--help`, add 2–3 concrete example invocations (local, CI, different warehouses). [fuchsia](https://fuchsia.dev/fuchsia-src/development/api/cli_help) - -## 2. Document the “golden workflows” - -Write 3 short “How to use” flows in README, with copy‑paste commands: - -1. Local check on existing stack: - - `pip install pipelineprobe` - - `pipelineprobe init` - - Edit YAML with Airflow URL, dbt paths, Postgres DSN. - - `pipelineprobe audit --format html` → open report. - -2. CI usage (GitHub Actions): - - Full example workflow that runs `pipelineprobe audit` on schedule and uploads HTML as an artifact. - - Show how `fail_on_critical` gates merges. - -3. Consulting / one‑off audit: - - “Clone client repo / connect to their Airflow, run, send them the HTML report + your notes.” - - This positions it as a billable tool for you, not just OSS. - -These should match the core journeys described in good CLI design docs (usage + examples, not just API). [fuchsia](https://fuchsia.dev/fuchsia-src/development/api/cli_help) - -## 3. Tighten positioning vs other tools - -In README, add one short section “How PipelineProbe is different” referencing common open‑source data‑quality / observability tools (Great Expectations, Soda, dbt tests) as context. [decube](https://www.decube.io/post/why-apache-airflow-is-not-the-best-tool-for-data-quality-checks) - -Small table: - -- Column: “Tool”, “What it focuses on”, “Where PipelineProbe fits”. -- Emphasise: “Point‑in‑time infra audit on top of Airflow + dbt + warehouse; read‑only, zero code change”. [willowvibe-web.vercel](https://willowvibe-web.vercel.app) - -This makes it clear you’re not competing directly with full observability stacks, but giving a quick audit lens. - -## 4. Improve CLI ergonomics - -A couple of small but high‑impact changes: - -1. Add `--version` and `pipelineprobe --help` output examples in README. [github](https://github.com/arturtamborski/cli-best-practices) -2. Add a `pipelineprobe doctor` (future, can stub now): - - Validates connectivity to Airflow/dbt/warehouse and prints “what will be checked” without running full rules. -3. Exit codes: - - Already: non‑zero when `critical_count > fail_on_critical`. - - Document the mapping (0 OK, 1 threshold breached, maybe 2 config error) so teams can wire it into CI policies. [github](https://github.com/arturtamborski/cli-best-practices) - -## 5. Add “marketing‑grade” output for real usage - -Your HTML report now looks good; push it over the line as client‑ready: - -- Add one small section summarizing: - - “Top 3 actions to take this week” – choose the first 3 `critical`/`warning` issues sorted by severity and maybe category. -- Include environment metadata at the top: - - Airflow base URL (obfuscated host), warehouse type, and dbt target name (already in config; just pass into template). -- Add a “generated with `pipelineprobe vX.Y.Z`” footer, to reinforce the tool name. - -This turns the report into something you can screenshot in blog posts and client decks. - -## 6. Release hygiene - -Before calling it “practically usable” for strangers: - -1. Tag a `v0.1.0` GitHub release. -2. Publish to PyPI so `pip install pipelineprobe` works. -3. Add a short “Roadmap” section: next items could be: - - Dagster/Prefect connector - - BigQuery/Snowflake‑specific warehouse rules - - Basic cost insights (top tables by scanned bytes where available). [atlan](https://atlan.com/open-source-data-quality-tools/) - -That gives users confidence it’s maintained and lets you talk about it publicly (LinkedIn, Reddit, r/dataengineering, etc.) with a clean story. - -If you want, next step we can design that quickstart `docker-compose.yml` plus a tiny dbt example so someone can get from zero to a working HTML report on their laptop with copy‑paste only. \ No newline at end of file +# Bugs & Improvements Tracker + +This file tracks known issues, planned improvements, and the current state of each item. + +--- + +## Completed + +### Report UI +- [x] Add "Top 3 actions to take" summary section — implemented in renderer and template. +- [x] Include environment metadata (Airflow URL, warehouse type, dbt target) in the report header. +- [x] Add a "generated with PipelineProbe vX.Y.Z" footer. +- [x] Add an Info count card alongside Critical and Warnings in the summary grid. +- [x] SVG animated health-score ring replacing the plain numeric display. +- [x] Severity filter buttons (All / Critical / Warnings / Info) on the findings list. +- [x] Affected resource tags shown on each finding card. +- [x] Subtle per-severity background tints on issue cards. +- [x] Responsive layout and print CSS. + +### CLI & DX +- [x] `pipelineprobe doctor` command — validates connectivity to Airflow, dbt artifacts, and the warehouse. +- [x] `pipelineprobe diff` command — compares two JSON reports and surfaces regressions / improvements with coloured output and exit code 1 on regression. +- [x] `--version` flag. +- [x] `--format` validation with clear error on unsupported values. +- [x] `--fail-on-critical` documented in README and CI guide. +- [x] Exit codes documented (`0` = success, `1` = threshold breached or error). + +### Bug Fixes +- [x] Airflow connector pagination — organisations with >100 DAGs no longer receive silently truncated results. +- [x] `check_stale_dags` — correctly flags active DAGs with zero recorded runs. +- [x] `BigQueryConnector.has_timestamps` — was hardcoded `True`; now queries `INFORMATION_SCHEMA.COLUMNS`. +- [x] `SnowflakeConnector` — correlated subquery replaced with a CTE; empty-credential early return added. +- [x] `renderer.render_json()` — `TypeError` on `timedelta` serialization fixed. +- [x] All bare `print()` replaced with `logging.getLogger(__name__)`. +- [x] PostgreSQL connector uses `try/finally` to guarantee connection closure. + +### Docs +- [x] README overhauled: quick start, CLI reference table, exit codes, diff workflow, comparison table. +- [x] CONTRIBUTING.md updated: template variable reference, rule/connector extension guide. +- [x] docs/architecture.md updated: diff command, ring_offset, health score formula. +- [x] docs/configuration.md updated: `rules` section, `severity_overrides` table, exit codes. +- [x] docs/ci-integration.md updated: `diff` regression detection workflow, threshold guide, exit codes. +- [x] CHANGELOG.md: proper version comparison links added. + +--- + +## Open / Planned + +### v0.2.0 — Connectors +- [ ] Prefect connector. +- [ ] Dagster connector. + +### v0.3.0 — Cost Insights +- [ ] BigQuery: top tables by scanned bytes. +- [ ] Snowflake: credit consumption per warehouse. + +### v1.0.0 — Lineage +- [ ] Basic data lineage support (upstream/downstream DAG relationships). + +### Quickstart +- [ ] Harden the Docker Compose quickstart: add a minimal toy dbt project with models and `run_results.json` so new users get a fully populated HTML report out of the box. + +### Release Hygiene +- [ ] Tag `v0.1.0` GitHub release. +- [ ] Publish to PyPI so `pip install pipelineprobe` resolves to the correct package. diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 198df28..3a6d364 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -1,14 +1,17 @@ # CI/CD Integration -PipelineProbe is designed to run seamlessly in your continuous integration and deployment pipelines. By failing your build when regressions are identified, PipelineProbe acts as a quality gate for your data platform. +PipelineProbe is designed to run seamlessly in your continuous integration pipelines. By failing a build when regressions are detected, it acts as an automated quality gate for your data platform. --- ## GitHub Actions -We provide a ready-to-use workflow at [`examples/github-actions/pipelineprobe.yml`](../examples/github-actions/pipelineprobe.yml). The core audit step: +A ready-to-use workflow is available at [`examples/github-actions/pipelineprobe.yml`](../examples/github-actions/pipelineprobe.yml). The core audit step: ```yaml + - name: Install PipelineProbe + run: pip install pipelineprobe + - name: Run PipelineProbe Audit env: PIPELINEPROBE_AIRFLOW_PASSWORD: ${{ secrets.AIRFLOW_PASSWORD }} @@ -20,9 +23,9 @@ We provide a ready-to-use workflow at [`examples/github-actions/pipelineprobe.ym --fail-on-critical 5 ``` -### Archive Reports +### Uploading Reports as Artifacts -We strongly recommend uploading the generated reports so they are accessible per-run: +Always upload the generated reports so they are accessible per run: ```yaml - name: Upload PipelineProbe Reports @@ -34,6 +37,33 @@ We strongly recommend uploading the generated reports so they are accessible per retention-days: 30 ``` +### Regression Detection with `diff` + +Track quality trends by comparing the current report against a stored baseline: + +```yaml + - name: Download baseline report + uses: actions/download-artifact@v4 + with: + name: pipelineprobe-baseline + path: baseline/ + + - name: Run Audit + run: pipelineprobe audit --format json + + - name: Diff against baseline + run: pipelineprobe diff baseline/report.json reports/report.json + + - name: Promote current report to baseline + if: success() + uses: actions/upload-artifact@v4 + with: + name: pipelineprobe-baseline + path: reports/report.json +``` + +`pipelineprobe diff` exits with code `1` if any new issues appear in the current report, making regressions immediately visible in CI. + --- ## GitLab CI @@ -64,40 +94,62 @@ Always inject sensitive values via your CI provider's secret management — **ne | Environment Variable | Purpose | |---|---| | `PIPELINEPROBE_AIRFLOW_PASSWORD` | Airflow REST API password | -| `PIPELINEPROBE_WAREHOUSE_DSN` | PostgreSQL DSN | -| `GOOGLE_APPLICATION_CREDENTIALS` | Path to BigQuery service account JSON | +| `PIPELINEPROBE_WAREHOUSE_DSN` | PostgreSQL connection DSN | +| `GOOGLE_APPLICATION_CREDENTIALS` | Path to BigQuery service account JSON key | -For BigQuery, mount the service account key as a CI secret file and set `GOOGLE_APPLICATION_CREDENTIALS` to the mounted path. +**BigQuery** — Mount the service account key as a CI secret file and set `GOOGLE_APPLICATION_CREDENTIALS` to its path. -For Snowflake, set `warehouse.account`, `username`, and `password` in `pipelineprobe.yml` and inject the password via a custom env var that you reference in the YAML, or configure via a secrets manager. +**Snowflake** — Set `warehouse.account` and `username` in the YAML; inject the password via a CI secret or a secrets manager reference. Do not store the password in the YAML file. --- -## Failing the Build (`fail_on_critical`) +## `fail_on_critical` — Build Quality Gate Configure PipelineProbe to return exit code `1` when critical issues exceed a threshold: ```yaml +# pipelineprobe.yml report: fail_on_critical: 0 # fail if even 1 critical issue is found ``` -- `0` = fail on **any** critical issue (strictest, ideal for greenfield projects) -- `5` = tolerate up to 5 critical issues (useful when onboarding to legacy stacks) - -The threshold can also be overridden per-run without changing the config: +Or override per run without editing the config file: ```bash pipelineprobe audit --fail-on-critical 0 ``` +### Threshold guide + +| Setting | When to use | +|---|---| +| `0` | Greenfield or high-standards teams — fail on any critical issue | +| `5` | Onboarding to a legacy stack — tolerate a small backlog while teams address issues | +| `50` | Initial adoption — capture a baseline without blocking CI immediately | + --- ## Gradually Tightening Standards -A recommended adoption pattern for existing codebases: +A recommended adoption pattern: + +1. Start with `fail_on_critical: 50` to avoid blocking the team on day one. +2. Each sprint, reduce the threshold by the number of issues your team resolved. +3. Archive `report.json` each run and use `pipelineprobe diff` to track progress. +4. Aim for `fail_on_critical: 0` as the long-term standard. -1. Start with `fail_on_critical: 50` to ensure the CI job doesn't block your team immediately. -2. Each sprint, reduce the threshold by however many issues you fix. -3. Track progress by archiving `report.json` and comparing counts over time. +--- + +## Exit Codes + +| Code | Meaning | +|---|---| +| `0` | Audit succeeded; critical count at or below threshold | +| `1` | Critical count exceeds threshold, or a config / connectivity error | + +The `diff` command follows the same convention: +| Code | Meaning | +|---|---| +| `0` | No regressions detected between the two reports | +| `1` | One or more new issues appeared in the current report | diff --git a/docs/configuration.md b/docs/configuration.md index bc775fd..8ecead2 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,6 +1,6 @@ # Configuration Reference -This document describes every configuration option supported by PipelineProbe, including the `pipelineprobe.yml` YAML fields and recognised environment variables. +This document describes every configuration option supported by PipelineProbe, including `pipelineprobe.yml` YAML fields, environment variable overrides, and CLI flags. --- @@ -28,9 +28,13 @@ report: output_dir: "./reports" format: "both" fail_on_critical: 5 + +rules: + stale_threshold_days: 7 + fetch_concurrency: 10 ``` -> **Tip**: Run `pipelineprobe init` to generate this file with defaults in your current directory. +> **Tip** — Run `pipelineprobe init` to generate this file with defaults in your current directory. --- @@ -43,9 +47,9 @@ report: | `username` | string | `"admin"` | Airflow username. | | `password` | string | `"admin"` | Airflow password. **Prefer the env var** `PIPELINEPROBE_AIRFLOW_PASSWORD`. | | `verify_ssl` | boolean | `false` | Whether to verify TLS certificates. Set to `true` in production. | -| `lookback_days` | integer | `14` | Number of days of DAG run history to fetch for analysis. | +| `lookback_days` | integer | `14` | Days of DAG run history to fetch for analysis. | -### Environment Variable Override +### Environment Variable | Variable | Overrides | |---|---| @@ -58,17 +62,17 @@ report: | Field | Type | Default | Description | |---|---|---|---| | `project_dir` | string | `"./analytics"` | Root directory of the dbt project. | -| `target` | string | `"prod"` | dbt target name (informational only, used for labelling). | -| `manifest_path` | string | `"target/manifest.json"` | Path to `manifest.json` **relative to `project_dir`**. | +| `target` | string | `"prod"` | dbt target name (informational — used for report labelling). | +| `manifest_path` | string | `"target/manifest.json"` | Path to `manifest.json` relative to `project_dir`. | | `run_results_path` | string | `"target/run_results.json"` | Path to `run_results.json` relative to `project_dir`. | -> **Note**: If `manifest.json` does not exist, the dbt connector is skipped gracefully and the report will note the absence. No error is raised. +> **Note** — If `manifest.json` does not exist, the dbt connector is skipped gracefully and the report will note its absence. No error is raised. --- ## `warehouse` — Warehouse Connection -The `type` field selects which connector is used. +The `type` field selects which connector is used. Each warehouse type has its own set of required fields. ### PostgreSQL @@ -81,7 +85,7 @@ warehouse: | Field | Description | |---|---| | `type` | Must be `"postgres"` | -| `dsn` | A full PostgreSQL DSN string. **Prefer the env var** `PIPELINEPROBE_WAREHOUSE_DSN`. | +| `dsn` | Full PostgreSQL DSN string. **Prefer the env var** `PIPELINEPROBE_WAREHOUSE_DSN`. | ### BigQuery @@ -94,9 +98,9 @@ warehouse: | Field | Description | |---|---| | `type` | Must be `"bigquery"` | -| `project_id` | GCP project ID. If omitted, the default project from `GOOGLE_APPLICATION_CREDENTIALS` or ADC is used. | +| `project_id` | GCP project ID. If omitted, falls back to the default project from ADC. | -Authentication is handled via [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). Set the `GOOGLE_APPLICATION_CREDENTIALS` env var to the path of a service account JSON key. +Authentication is handled via [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials). Set `GOOGLE_APPLICATION_CREDENTIALS` to the path of a service account JSON key, or use `gcloud auth application-default login` locally. ### Snowflake @@ -113,13 +117,14 @@ warehouse: | `type` | Must be `"snowflake"` | | `account` | Snowflake account identifier (e.g. `xyz.us-east-1`). | | `username` | Snowflake user. | -| `password` | Snowflake password. | +| `password` | Snowflake password. Store in a secret manager or CI secret rather than in the YAML. | -### Environment Variable Override +### Environment Variables | Variable | Overrides | |---|---| | `PIPELINEPROBE_WAREHOUSE_DSN` | `warehouse.dsn` (PostgreSQL only) | +| `GOOGLE_APPLICATION_CREDENTIALS` | Path to BigQuery service account JSON key | --- @@ -128,9 +133,43 @@ warehouse: | Field | Type | Default | Description | |---|---|---|---| | `output_dir` | string | `"./reports"` | Directory where reports are written. Created automatically if missing. | -| `format` | string | `"html"` | Output format: `html`, `json`, or `both`. Can be overridden via `--format` CLI flag. | +| `format` | string | `"html"` | Output format: `html`, `json`, or `both`. Can be overridden with `--format`. | | `include_cost_section` | boolean | `false` | Reserved for a future cost analysis section. Has no effect in v0.1. | -| `fail_on_critical` | integer | `5` | Non-zero exit code is returned if the number of critical issues **exceeds** this value. Set to `0` to fail on any single critical issue. Can be overridden via `--fail-on-critical` CLI flag. | +| `fail_on_critical` | integer | `5` | Return exit code `1` when critical issue count **exceeds** this value. Set to `0` to fail on any single critical issue. Can be overridden with `--fail-on-critical`. | + +--- + +## `rules` — Rule Engine Settings + +| Field | Type | Default | Description | +|---|---|---|---| +| `stale_threshold_days` | integer | `7` | Number of days without a successful DAG run before it is flagged as stale. | +| `fetch_concurrency` | integer | `10` | Maximum concurrent Airflow API calls when fetching DAG run history and task configs. | +| `severity_overrides` | dict | `{}` | Per-rule severity overrides. See table below. | + +### `severity_overrides` + +Override the default severity of any built-in rule. Valid values are `critical`, `warning`, or `info`. + +```yaml +rules: + severity_overrides: + missing_sla: critical # default: info — raise to critical for SLA-sensitive teams + missing_retries: warning # default: warning — no change needed + stale_dags: critical # default: warning — raise to critical for prod monitors + high_failure_rate: critical # default: critical +``` + +| Rule key | Default severity | Description | +|---|---|---| +| `missing_retries` | `warning` | Tasks with zero retries configured | +| `missing_slas` | `info` | Tasks without an SLA timeout | +| `high_failure_rate` | `critical` | DAGs with >20% failure rate over ≥5 runs | +| `stale_dags` | `warning` | Active DAGs with no successful run in `stale_threshold_days` | +| `missing_tests` | `warning` | dbt models with zero tests | +| `failing_models` | `critical` | dbt models that failed their last run | +| `large_tables` | `warning` | Tables with >10M rows | +| `missing_timestamps` | `warning` | Tables >1M rows without `created_at` / `updated_at` | --- @@ -147,15 +186,22 @@ pipelineprobe audit \ | Flag | Equivalent Config | Description | |---|---|---| -| `--config` | N/A | Path to the YAML config file. | -| `--format` | `report.format` | Override the report output format. | -| `--fail-on-critical` | `report.fail_on_critical` | Override the critical issue threshold. | +| `--config FILE` | — | Path to the YAML config file. | +| `--format FORMAT` | `report.format` | Override report format: `html`, `json`, or `both`. | +| `--fail-on-critical N` | `report.fail_on_critical` | Override the critical issue threshold. | + +### Exit Codes + +| Code | Meaning | +|---|---| +| `0` | Audit completed; critical count at or below `fail_on_critical` | +| `1` | Critical count exceeds threshold, or a config / connectivity error occurred | --- ## Security Best Practices -1. **Never store passwords in `pipelineprobe.yml` when committing to source control.** Use environment variables instead. -2. `.pipelineprobe.yml` is listed in `.gitignore` by default. Verify this is in place. -3. PipelineProbe is **read-only** — it never writes to Airflow, dbt, or your warehouse. -4. DSN strings and passwords are never written to report output. +1. **Never store passwords in `pipelineprobe.yml` when committing to source control.** Use environment variables or a secrets manager instead. +2. `pipelineprobe.yml` is listed in `.gitignore` by default — verify it is in place before your first commit. +3. PipelineProbe is **strictly read-only** — it never writes to Airflow, dbt, or your warehouse. +4. DSN strings and passwords are never written to any report output (HTML or JSON). diff --git a/pipelineprobe/cli.py b/pipelineprobe/cli.py index 97a54c0..c0df6f5 100644 --- a/pipelineprobe/cli.py +++ b/pipelineprobe/cli.py @@ -139,6 +139,7 @@ def audit( dag_count = max(1, len(airflow_dags)) critical_count = sum(1 for i in issues if i.severity == "critical") warning_count = sum(1 for i in issues if i.severity == "warning") + info_count = sum(1 for i in issues if i.severity == "info") critical_density = critical_count / dag_count warning_density = warning_count / dag_count @@ -151,6 +152,7 @@ def audit( "score": score, "critical_count": critical_count, "warning_count": warning_count, + "info_count": info_count, "total_issues": len(issues), "dag_count": dag_count, "score_formula": ( diff --git a/pipelineprobe/renderer.py b/pipelineprobe/renderer.py index a8a968e..345c5be 100644 --- a/pipelineprobe/renderer.py +++ b/pipelineprobe/renderer.py @@ -27,13 +27,19 @@ def render_html(self, issues: list[Issue], summary: dict) -> Path: template = self.env.get_template("report.html") generated_at = datetime.now().strftime("%Y-%m-%d %H:%M") - - # Identify top 3 critical/warning actions + + # Identify top 3 critical/warning actions sorted by severity then category top_actions = sorted( [i for i in issues if i.severity in ("critical", "warning")], key=lambda x: (x.severity == "warning", x.category), )[:3] + # SVG ring offset: circumference = 2π × r(40) ≈ 251.3 + # offset = circumference × (1 − score/100) → 0 means full ring, 251.3 means empty + _circ = 251.3 + score = summary.get("score", 0) + ring_offset = round(_circ * (1 - score / 100), 2) + html_content = template.render( issues=issues, summary=summary, @@ -41,6 +47,7 @@ def render_html(self, issues: list[Issue], summary: dict) -> Path: version=__version__, top_actions=top_actions, metadata=summary.get("metadata", {}), + ring_offset=ring_offset, ) output_path = self.output_dir / "pipelineprobe-report.html" with open(output_path, "w") as f: diff --git a/pipelineprobe/templates/report.html b/pipelineprobe/templates/report.html index 5ccc6b7..b2a1fbc 100644 --- a/pipelineprobe/templates/report.html +++ b/pipelineprobe/templates/report.html @@ -4,198 +4,594 @@ PipelineProbe Audit Report - + -
-
-
-

PipelineProbe

-

Data Infrastructure Audit Engine

-
-
- Report generated on {{ generated_at }} + + +
+
+
+
PP
+ PipelineProbe
-
+ Generated {{ generated_at }} +
+ + +
+ + +
+

Audit Report

+

Point-in-time infrastructure audit — {{ summary.dag_count }} DAG{% if summary.dag_count != 1 %}s{% endif %} analysed

+
+
+ +
- Health Score - {{ summary.score }} - out of 100 + Health Score +
+ + + + + {{ summary.score }} +
+ out of 100
-
- Critical Issues -
{{ summary.critical_count }}
+ + +
+
+ Critical Issues +
🔴
+
+
{{ summary.critical_count }}
-
- Warnings -
{{ summary.warning_count }}
+ + +
+
+ Warnings +
⚠️
+
+
{{ summary.warning_count }}
+ + +
+
+ Info +
ℹ️
+
+
{{ summary.info_count }}
+
+
- {% if top_actions %} -

Top Actions to Take

-
- {% for action in top_actions %} -
-
- {{ action.category|upper }} - {{ action.severity }} + +
+
+

+ + Environment +

+
+
+
+
+ Orchestrator + {{ metadata.orchestrator_url }} +
+
+ Warehouse + {{ metadata.warehouse_type | capitalize }} +
+
+ dbt Target + {{ metadata.dbt_target }} +
+
+ Audit Mode + point-in-time +
-
{{ action.summary }}
-
- Action: {{ action.recommendation }} +
+
+ + + {% if top_actions %} +
+
+

+ + Top Actions +

+
+
+ {% for action in top_actions %} +
+
{{ loop.index }}
+
+
+ {{ action.severity }} + {{ action.category }} +
+
{{ action.summary }}
+
{{ action.recommendation }}
+
+ {% endfor %}
- {% endfor %}
{% endif %} -

Environment Metadata

-
-
Orchestrator: {{ metadata.orchestrator_url }}
-
Warehouse: {{ metadata.warehouse_type|capitalize }}
-
dbt Target: {{ metadata.dbt_target }}
-
Mode: Point-in-time Audit
-
+ +
+
+

+ + All Findings + ({{ issues | length }}) +

+
+ + + + +
+
-

Findings Breakdown

-
- {% for issue in issues %} -
-
-
- {{ issue.category|upper }} -
{{ issue.summary }}
+
+ {% for issue in issues %} +
+
+
+
+ {{ issue.severity }} + {{ issue.category }} +
+
{{ issue.summary }}
+
+
+
{{ issue.details }}
+
+ Recommendation + {{ issue.recommendation }} +
+ {% if issue.affected_resources %} +
+ {% for res in issue.affected_resources %} + {{ res }} + {% endfor %}
- {{ issue.severity }} + {% endif %}
-
{{ issue.details }}
-
- Recommendation - {{ issue.recommendation }} + {% else %} +
+
🎉
+
All clear!
+

No issues found. Your data infrastructure follows all evaluated best practices.

+ {% endfor %}
- {% else %} -
-
🎉
-
No issues identified!
-

Your data infrastructure is following all evaluated best practices.

-
- {% endfor %}
+
- Generated by PipelineProbe v{{ version }} · Built by - WillowVibe + Generated by PipelineProbe v{{ version }} + Built by WillowVibe
-
+ +
+ +