Skip to content

Add golden dataset & evaluation framework for agent performance (#308) - #350

Open
sudo-ss1 wants to merge 2 commits into
interviewstreet:mainfrom
sudo-ss1:feat/golden-dataset-eval
Open

Add golden dataset & evaluation framework for agent performance (#308)#350
sudo-ss1 wants to merge 2 commits into
interviewstreet:mainfrom
sudo-ss1:feat/golden-dataset-eval

Conversation

@sudo-ss1

Copy link
Copy Markdown

Add golden dataset & evaluation framework for agent performance (#308)

Closes #308.

What & why

Adds a golden dataset of labeled résumés plus an evaluation runner so the agent's
scoring can be measured and compared across models (the issue's motivation:
understanding performance when swapping in ad-hoc models).

It measures at the evaluator boundary: each case supplies a structured
résumé (JSONResume) + optional GitHub data, and the runner compares the
evaluator's output against human-assigned bands and per-category tolerance
ranges
. This isolates scoring quality (what changes when you swap models) from
PDF parsing, extraction, and live GitHub calls. All committed résumés are
synthetic (no PII, no real GitHub accounts).

What's included

  • evals/golden/*.json — 9 synthetic labeled résumés (3 strong / 3 medium /
    3 weak, incl. edge cases: OSS-only, production-only, tutorial-projects-only).
  • evals/run_eval.py — runs each case through the real evaluator; reports band
    accuracy, per-category within-range rate + bias (calibration), Spearman rank
    correlation, and cross-repeat stability; writes a model-stamped JSON report.
    Flags: --repeat, --filter, --out, --min-band-accuracy (regression gate).
  • evals/metrics.py — pure, offline metric functions.
  • evals/tests/ — offline unit + smoke + dataset-integrity tests (no LLM/network).
  • score_utils.py — extracted score aggregation so the CLI and the eval compute
    the total identically (behavior-preserving refactor of score.py).
  • README updates + evals/README.md (format, metrics, how to add cases).

Usage

python -m evals.run_eval                 # score the dataset, print metrics
python -m evals.run_eval --repeat 3      # average runs to smooth LLM noise
LLM_PROVIDER=gemini DEFAULT_MODEL=gemini-2.0-flash python -m evals.run_eval
python -m pytest evals/tests             # offline tests, no backend needed

Testing

  • 54 offline tests pass; Black clean.
  • Refactor verified behavior-preserving (CLI still prints the same total and cap
    warning for a sample evaluation).
  • Not run against a live LLM in CI (requires an Ollama/Gemini backend); the tests
    run fully offline via a dependency-injected evaluator.

Notes for reviewers

  • Follows CONTRIBUTING: Black-formatted, offline smoke tests exercising each
    stage, issue referenced. No prompt changes, so no before/after prompt examples.
  • The seed set is intentionally small and designed to be grown by contributors;
    evals/README.md documents the case format and how to add more.

sudo-ss1 added 2 commits July 15, 2026 02:47
Move the total-score aggregation out of score.print_evaluation_results into
a dependency-light score_utils.aggregate_scores so the CLI and the new evals
framework compute the final score identically. Behavior is preserved.
Add evals/ with a golden-case JSON format (synthetic resume + human
bands/ranges), a seed dataset of 9 labeled synthetic resumes across
strong/medium/weak, a runner that scores each case via the real evaluator
and reports band accuracy, per-category within-range + bias, Spearman rank
correlation and cross-repeat stability, pure offline metric functions, and
offline unit/smoke/dataset-integrity tests (no LLM required).

Comparing models is just an LLM_PROVIDER/DEFAULT_MODEL change.
@sudo-ss1
sudo-ss1 force-pushed the feat/golden-dataset-eval branch from 91d3a23 to 58096f7 Compare July 15, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Create a golden dataset for evaluation of the agent’s performance

1 participant