Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,18 @@

All notable changes to PipelineProbe are documented here.

The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
Versioning follows [Semantic Versioning](https://semver.org/).

---

## [Unreleased]

### Added
- **Report UI overhaul** — sticky topbar, animated SVG health-score ring, severity filter buttons (All / Critical / Warnings / Info), affected-resource tags on each finding card, subtle per-severity background tints, and responsive print styles.
- **Info count card** — the summary grid now shows a fourth card for informational findings alongside Critical and Warnings.
- `info_count` included in the JSON report summary for downstream tooling.
- `ring_offset` computed in the renderer and injected into the Jinja2 template so the SVG ring accurately reflects the health score.
- BigQuery connector querying `INFORMATION_SCHEMA.TABLE_STORAGE` and `COLUMNS` for real `has_timestamps` detection.
- Snowflake connector using a CTE pattern to work around correlated subquery restrictions.
- `python-dateutil` declared as an explicit dependency in `pyproject.toml`.
Expand All @@ -34,12 +38,12 @@ Versioning follows [Semantic Versioning](https://semver.org/).
## [0.1.0] — 2026-03-15

### Added
- Phase 0: Project skeleton, `pyproject.toml`, CI workflow (ruff + pytest), CLI entry point.
- Phase 1 (MVP connectors):
- **Phase 0** — Project skeleton, `pyproject.toml`, CI workflow (ruff + pytest), CLI entry point.
- **Phase 1 MVP connectors:**
- `AirflowConnector` — fetches DAGs, DAG runs, and task configurations via the Airflow REST API.
- `DbtConnector` — reads `manifest.json` and `run_results.json`; counts tests per model.
- `PostgresConnector` — queries `pg_stat_user_tables` and `information_schema.columns`.
- Phase 1 (Rules Engine):
- **Phase 1 Rules Engine:**
- `check_missing_retries` — warns on tasks with no retry configuration.
- `check_missing_slas` — informs on tasks with no SLA.
- `check_high_failure_rate` — critical alert for DAGs with >20% failure rate over ≥5 runs.
Expand All @@ -48,13 +52,20 @@ Versioning follows [Semantic Versioning](https://semver.org/).
- `check_failing_models` — critical alert for dbt models that failed their last run.
- `check_large_tables` — warns on tables with >10M rows.
- `check_missing_timestamps` — warns on tables >1M rows without `created_at`/`updated_at`.
- Phase 1 (Report Renderer): Jinja2 HTML template; JSON output via `model_dump()`.
- Phase 2 (DX & CI):
- **Phase 1 Report Renderer** — Jinja2 HTML template; JSON output via `model_dump()`.
- **Phase 2 DX & CI:**
- `pipelineprobe init` command generating a default `pipelineprobe.yml`.
- `--fail-on-critical` and `--format` CLI overrides.
- `examples/github_action.yml` GitHub Actions workflow.
- `examples/github-actions/pipelineprobe.yml` GitHub Actions workflow.
- `docs/ci-integration.md`.
- `.gitignore` with standard Python exclusions.
- Phase 3 (Extended Integrations):
- **Phase 3 Extended Integrations:**
- `BigQueryConnector`.
- `SnowflakeConnector`.
- `pipelineprobe doctor` command for connectivity validation.
- `pipelineprobe diff` command for regression detection between two JSON reports.

---

[Unreleased]: https://github.com/willowvibe/pipelineprobe/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/willowvibe/pipelineprobe/releases/tag/v0.1.0
110 changes: 73 additions & 37 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing to PipelineProbe

Thank you for your interest in contributing! PipelineProbe is an open-source project by [WillowVibe](https://www.willowvibe.com) and we welcome contributions of all kinds, including bug reports, documentation improvements, and new connectors.
Thank you for your interest in contributing! PipelineProbe is an open-source project by [WillowVibe](https://www.willowvibe.com) and we welcome contributions of all kinds bug reports, documentation improvements, new rules, and new connectors.

---

Expand All @@ -18,12 +18,10 @@ Thank you for your interest in contributing! PipelineProbe is an open-source pro
git clone https://github.com/willowvibe/pipelineprobe.git
cd pipelineprobe

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate # Linux / macOS
# .venv\Scripts\activate # Windows

# Install in editable mode with all dev dependencies
pip install -e ".[dev]"
```

Expand All @@ -35,21 +33,28 @@ pip install -e ".[dev]"
pytest
```

Tests live in the `tests/` directory and use `pytest`. Fixtures for mock data are in `tests/conftest.py`.
Tests live in `tests/` and use `pytest`. Fixtures for mock data are in `tests/conftest.py`.

To run a specific file or test:

```bash
pytest tests/test_rules.py
pytest tests/test_renderer.py::test_render_html_no_issues
```

---

## Linting
## Linting & Formatting

We use [ruff](https://docs.astral.sh/ruff/) for linting and formatting:
We use [ruff](https://docs.astral.sh/ruff/) for both linting and formatting:

```bash
ruff check . # Check for lint errors
ruff check --fix . # Auto-fix safe issues
ruff format . # Format code
ruff check . # report lint errors
ruff check --fix . # auto-fix safe issues
ruff format . # format all files
```

CI will fail if `ruff check .` reports any errors.
CI fails if `ruff check .` reports any errors. Run both before opening a PR.

---

Expand All @@ -58,24 +63,30 @@ CI will fail if `ruff check .` reports any errors.
```
pipelineprobe/
├── pipelineprobe/
│ ├── cli.py # CLI entry points (audit, init)
│ ├── config.py # YAML + env var config loader (Pydantic)
│ ├── models.py # Core domain models (Dag, Task, DbtModel, Issue)
│ ├── renderer.py # HTML + JSON report renderer (Jinja2)
│ ├── cli.py # CLI entry points (audit, init, doctor, diff)
│ ├── config.py # YAML + env var config loader (Pydantic)
│ ├── models.py # Core domain models (Dag, Task, DbtModel, Issue)
│ ├── renderer.py # HTML + JSON report renderer (Jinja2)
│ ├── connectors/
│ │ ├── airflow.py # Apache Airflow REST API connector
│ │ ├── dbt.py # dbt manifest / run_results reader
│ │ ├── postgres.py # PostgreSQL INFORMATION_SCHEMA queries
│ │ ├── bigquery.py # BigQuery INFORMATION_SCHEMA queries
│ │ └── snowflake.py # Snowflake INFORMATION_SCHEMA queries
│ │ ├── airflow.py # Apache Airflow REST API connector
│ │ ├── dbt.py # dbt manifest / run_results reader
│ │ ├── postgres.py # PostgreSQL INFORMATION_SCHEMA queries
│ │ ├── bigquery.py # BigQuery INFORMATION_SCHEMA queries
│ │ └── snowflake.py # Snowflake INFORMATION_SCHEMA queries
│ ├── rules/
│ │ ├── engine.py # Pluggable rule registration and execution
│ │ ├── engine.py # Pluggable rule registration and execution
│ │ ├── airflow_rules.py
│ │ ├── dbt_rules.py
│ │ └── postgres_rules.py
│ └── templates/
│ └── report.html # Jinja2 HTML report template
│ └── report.html # Jinja2 HTML report template
├── tests/
│ ├── conftest.py # Shared pytest fixtures
│ ├── test_cli.py
│ ├── test_config.py
│ ├── test_connectors.py
│ ├── test_renderer.py
│ └── test_rules.py
├── docs/
├── examples/
└── pyproject.toml
Expand All @@ -88,18 +99,20 @@ pipelineprobe/
1. Add a function to the appropriate `rules/` module (e.g. `airflow_rules.py`):

```python
from typing import List
from pipelineprobe.models import Issue

def check_my_new_rule(context: dict) -> List[Issue]:
issues = []
for dag in context.get("airflow_dags", []):
# Your logic here
if some_condition:
if some_condition(dag):
issues.append(Issue(
severity="warning",
category="dag",
summary="...",
details="...",
recommendation="...",
affected_resources=[dag.id]
summary="Short description of the problem",
details="Longer explanation with context.",
recommendation="Concrete steps to fix it.",
affected_resources=[dag.id],
))
return issues
```
Expand All @@ -112,39 +125,62 @@ def register_airflow_rules(engine):
engine.register_rule(check_my_new_rule)
```

3. Write a test in `tests/test_rules.py`.
3. Write tests in `tests/test_rules.py` — at minimum one happy-path and one edge-case test.

4. If the rule is configurable (e.g. severity override), document it in `docs/configuration.md` under the `rules.severity_overrides` table.

---

## Adding a New Connector

1. Create `pipelineprobe/connectors/my_connector.py`.
2. Implement a class with a `get_stats_sync() -> List[Dict[str, Any]]` method that returns rows with `schemaname`, `tablename`, `row_count`, and `has_timestamps`.
3. Add your new warehouse type to `cli.py` in the `if cfg.warehouse.type ==` block.
2. Implement a class with a `get_stats_sync() -> List[Dict[str, Any]]` method returning rows with at least: `schemaname`, `tablename`, `row_count`, `has_timestamps`.
3. Route `cfg.warehouse.type == "my_connector"` in the `audit()` function in `cli.py`.
4. Add any required credentials to `WarehouseConfig` in `config.py`.
5. Document it in `docs/configuration.md`.
5. Document the new connector and its config fields in `docs/configuration.md`.
6. Add at least one connectivity test in `tests/test_connectors.py`.

---

## Working on the HTML Report

The report template lives at `pipelineprobe/templates/report.html` and is rendered by `renderer.py` using Jinja2.

Variables available in the template:

| Variable | Type | Description |
|---|---|---|
| `issues` | `List[Issue]` | All findings from the rule engine |
| `summary` | `dict` | Health score, counts, dag_count, metadata |
| `top_actions` | `List[Issue]` | Top 3 critical/warning issues |
| `metadata` | `dict` | orchestrator_url, warehouse_type, dbt_target |
| `generated_at` | `str` | Formatted timestamp string |
| `version` | `str` | PipelineProbe version |
| `ring_offset` | `float` | SVG ring stroke-dashoffset (0 = full, 251.3 = empty) |

To preview a rendered report, run a real or mock audit and open `./reports/pipelineprobe-report.html` in a browser.

---

## Pull Request Process

1. Fork the repo and create your branch from `main`.
2. Write tests for your changes (aim for at least one happy-path and one edge-case test).
2. Write tests for your changes.
3. Ensure `ruff check .` and `pytest` both pass.
4. Open a Pull Request with a clear description of:
4. Open a PR with a description covering:
- **What** changed
- **Why** it's needed
- **Why** it is needed
- **How** to test it manually

---

## Reporting Bugs

Please open a GitHub Issue with:
Open a GitHub Issue with:
- PipelineProbe version (`pip show pipelineprobe`)
- Python version
- A description of the bug and what you expected to happen
- Sanitized config (no credentials) and any relevant log output
- A clear description of what happened vs. what you expected
- Sanitized config (no credentials) and relevant log output

---

Expand Down
Loading
Loading