Skip to content

Add a per-language leakage dashboard renderer over benchmark runs#1757

Open
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-575-per-language-leakage-dashboard
Open

Add a per-language leakage dashboard renderer over benchmark runs#1757
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-575-per-language-leakage-dashboard

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Description

Adds a per-language leakage dashboard over one or more benchmark runs so multilingual leakage trends, residual-PHI categories, and the worst labels are visible without exposing source text or report metadata. The dashboard produces both a self-contained sortable HTML report and deterministic companion JSON suitable for CI threshold checks.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature which changes existing behavior)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test addition/improvement

Changes Made

  • Aggregate leakage counts and trends for every wired PII language from benchmark reports or OM-403 heatmaps.
  • Render one privacy-safe panel per language plus a sortable summary table, residual category breakdowns, and deterministic worst-label rankings.
  • Emit stable JSON with per-language evidence, optional leakage thresholds, combined gate status, and violations.

Testing

  • Added tests that prove the feature works.
  • New and existing unit tests pass locally.
  • Tested synthetic multilingual reports, direct heatmaps, single-language metrics, threshold failures, invalid inputs, and PHI exclusion.

Commands run:

  • make format
  • make lint
  • make format-check
  • make type-check
  • .venv/bin/python -m pytest tests/ -q — 5,205 passed, 46 skipped

Documentation

  • Added Google-style docstrings to the new public functions and classes.
  • Updated the changelog (not required for this scoped feature PR).

Code Quality

  • Ran make format, make lint, and make format-check.
  • Performed a self-review of the complete diff.
  • Added no new warnings or dependencies.

Dependencies

  • No new dependencies.

Checklist

  • Read the contributing and repository guidelines.
  • Commit has a clear, descriptive message.
  • Branch is based on current origin/master.

Related Issues

Closes #932

Depends on #650, completed by #680.

Screenshots/Examples

The generated HTML is self-contained and includes a sortable language summary followed by detailed language panels. Tests verify that all wired languages are present and that source text, model names, fixture identifiers, and metadata cannot enter either artifact.

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P2 Medium labels Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a per-language leakage dashboard renderer over benchmark runs

1 participant