Skip to content

Add regression escape tracker dashboard#1758

Open
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-579-regression-escape-tracker-dashboard
Open

Add regression escape tracker dashboard#1758
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-579-regression-escape-tracker-dashboard

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Description

Adds a metric-only regression-escape tracker over ordered benchmark release history. It distinguishes regressions stopped by release gates from regressions that shipped, records the release and elapsed time when escaped regressions recover, and exposes deterministic JSON plus a self-contained HTML dashboard for release-readiness review.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Test addition/improvement

Changes Made

  • Compare each suite and flattened metric against the last shipped benchmark report, classifying regressions as gated, escaped, or recovered.
  • Add release-readiness JSON with open-escape counts, recovery timing, and deterministic recurring-regression hotspot ranking.
  • Add a self-contained HTML dashboard and privacy guards that omit source metadata and sanitize non-technical identifiers.

Testing

  • I have added tests that prove my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have tested this change with multiple suites, metric directions, gate outcomes, release sequences, and unsafe metadata.
  • .venv/bin/python -m pytest tests/ -q — 5196 passed, 55 skipped.

Documentation

  • I have updated the documentation accordingly.
  • I have added docstrings to new functions/classes.
  • I have updated the CHANGELOG.md.

No user-facing documentation or changelog update is required for this focused evaluation API addition.

Code Quality

  • I ran make format, make lint, and make format-check.
  • For Swift/OpenMedKit changes, I ran make format-swift and make lint-swift.
  • I have performed a self-review of my own code.
  • I have commented the code where the release-history behavior is non-obvious.
  • My changes generate no new warnings.
  • Scoped mypy and pre-commit checks pass.

Dependencies

  • I have not added any new dependencies.
  • OR I have added new dependencies and they are justified because: N/A

Checklist

  • I have read the contributing guidelines.
  • My commit has a clear, descriptive message.
  • I have organized the commit appropriately.

Related Issues

Closes #936

Screenshots/Examples

The dashboard output is covered by deterministic, balanced-HTML, self-contained-asset, and no-raw-PHI tests.

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P2 Medium labels Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a regression-escape tracker dashboard over benchmark history

1 participant