Skip to content

Add a benchmark harness for the conflict classifier #24

Description

@venkatapgummadi

Background

We have unit tests for ConflictClassifier, but no benchmark to track classifier latency or accuracy
on a fixed corpus.

What needs to be done

Add ai-sync/benchmarks/classifier_benchmark.py that loads tests/fixtures/conflicts.jsonl, runs
classification, and prints accuracy + p50/p95 latency.

Definition of done

  • Script runs with python -m ai_sync.benchmarks.classifier_benchmark
  • Output saved as Markdown table
  • Linked from docs/metrics-and-kpis.md
    Estimated effort: 3–4 hours · Skills needed: Python, basic ML evaluation

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions