Add golden dataset & evaluation framework for agent performance (#308) - #350
Open
sudo-ss1 wants to merge 2 commits into
Open
Add golden dataset & evaluation framework for agent performance (#308)#350sudo-ss1 wants to merge 2 commits into
sudo-ss1 wants to merge 2 commits into
Conversation
Move the total-score aggregation out of score.print_evaluation_results into a dependency-light score_utils.aggregate_scores so the CLI and the new evals framework compute the final score identically. Behavior is preserved.
Add evals/ with a golden-case JSON format (synthetic resume + human bands/ranges), a seed dataset of 9 labeled synthetic resumes across strong/medium/weak, a runner that scores each case via the real evaluator and reports band accuracy, per-category within-range + bias, Spearman rank correlation and cross-repeat stability, pure offline metric functions, and offline unit/smoke/dataset-integrity tests (no LLM required). Comparing models is just an LLM_PROVIDER/DEFAULT_MODEL change.
sudo-ss1
force-pushed
the
feat/golden-dataset-eval
branch
from
July 15, 2026 01:47
91d3a23 to
58096f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add golden dataset & evaluation framework for agent performance (#308)
Closes #308.
What & why
Adds a golden dataset of labeled résumés plus an evaluation runner so the agent's
scoring can be measured and compared across models (the issue's motivation:
understanding performance when swapping in ad-hoc models).
It measures at the evaluator boundary: each case supplies a structured
résumé (
JSONResume) + optional GitHub data, and the runner compares theevaluator's output against human-assigned bands and per-category tolerance
ranges. This isolates scoring quality (what changes when you swap models) from
PDF parsing, extraction, and live GitHub calls. All committed résumés are
synthetic (no PII, no real GitHub accounts).
What's included
evals/golden/*.json— 9 synthetic labeled résumés (3 strong / 3 medium /3 weak, incl. edge cases: OSS-only, production-only, tutorial-projects-only).
evals/run_eval.py— runs each case through the real evaluator; reports bandaccuracy, per-category within-range rate + bias (calibration), Spearman rank
correlation, and cross-repeat stability; writes a model-stamped JSON report.
Flags:
--repeat,--filter,--out,--min-band-accuracy(regression gate).evals/metrics.py— pure, offline metric functions.evals/tests/— offline unit + smoke + dataset-integrity tests (no LLM/network).score_utils.py— extracted score aggregation so the CLI and the eval computethe total identically (behavior-preserving refactor of
score.py).evals/README.md(format, metrics, how to add cases).Usage
Testing
warning for a sample evaluation).
run fully offline via a dependency-injected evaluator.
Notes for reviewers
stage, issue referenced. No prompt changes, so no before/after prompt examples.
evals/README.mddocuments the case format and how to add more.