Add async DAG fetching, health checks, report diffing, and rule config

[Sruthi-ng]

Summary

This PR significantly enhances PipelineProbe's audit capabilities and operational tooling by introducing concurrent Airflow API calls, comprehensive health diagnostics, report comparison functionality, and configurable rule severity overrides.

Key Changes

Async DAG Fetching & Performance

• Implemented fetch_dag_details() in AirflowConnector using asyncio with semaphore-based concurrency control to fetch DAG runs and tasks in parallel
• Replaced sequential API calls with concurrent requests, reducing audit time for large Airflow installations
• Added configurable fetch_concurrency parameter (default: 10) to prevent API overwhelming

Enhanced doctor Command

• Replaced stub implementation with real connectivity validation:
  • Airflow: Probes /api/v1/health endpoint, reports metadatabase and scheduler status
  • dbt: Validates presence of manifest.json (required) and run_results.json (optional)
  • Warehouse: Implements driver-specific connection tests for Postgres, BigQuery, and Snowflake
• Returns exit code 1 on failures, enabling CI/CD integration
• Provides detailed error messages and safe credential display

New diff Command

• Compares two JSON audit reports to identify regressions and improvements
• Fingerprints issues by severity + summary to detect changes across runs
• Displays score delta, new issues, resolved issues, and unchanged count
• Returns exit code 1 if regressions detected (useful for CI gates)

Rule Configuration & Severity Overrides

• Added RulesConfig class supporting:
  • Per-rule severity overrides (e.g., treat missing_sla as critical for fintech teams)
  • Configurable stale_threshold_days (default: 7)
  • Configurable fetch_concurrency (default: 10)
• Updated all rule functions to respect severity overrides via _severity() helper
• Injected rule config into rule engine context for stateless rule execution

Health Score Improvements

• Replaced naive linear scoring with density-based formula:
  • Normalizes critical/warning counts by DAG count (criticals per DAG)
  • Critical density penalty: min(90, density × 200)
  • Warning density penalty: min(20, density × 40)
  • Prevents small shops from appearing as healthy as large shops with same issue counts
• Added score_formula to summary output for transparency

Configuration & Documentation

• Updated default pipelineprobe.yml template with new rules section
• Added inline comments explaining health score formula and rule override examples
• Improved import organization (alphabetical, grouped by type)

Notable Implementation Details

• Async implementation uses httpx.AsyncClient with auth preservation across concurrent requests
• Semaphore prevents connection pool exhaustion on large DAG counts
• Health check gracefully handles missing optional artifacts (run_results.json) with warnings
• Report diffing uses coarse fingerprinting to intentionally surface rule text changes as new issues
• All new features maintain backward compatibility with existing configs

https://claude.ai/code/session_014GhFtopVtaMtygBYvZJ7WE