feat: append scored runs to a HF Dataset results ledger

[kaaloo]

Problem

Scoring reports currently live as local files (--output reports/foo.json) or stdout. CI and Scaleway batch jobs have no place to push results that downstream comparison tools or dashboards can consume. This is the "open the door to CI/CD or Scaleway batch" goal from the broader plan.

The team also wants run IDs to be generated during the benchmark flow, not hand-typed, so they do not become a source of confusion across runs. The natural generator is manifest sync: a benchmark run starts by syncing a fresh manifest, and the manifest's run_id is the canonical handle downstream.

Scope

Add an eval-transcript results push command that appends a structured row to a configured HF Dataset results repo, plus a read-side results list for ad-hoc inspection. The push flow is designed to be the "last step" of a CI/batch job: benchmark pull → transcribe (one or more providers) → score all --output report.json → results push --scoring-report report.json.

Run ID generation

• manifest sync writes a new run_id field into data/manifest.md frontmatter on every invocation. New runs overwrite the previous one; users who want to keep multiple manifests in the same repo should use git.
• Format: <UTC-ISO-date>-<6-char-base32>, e.g. 2026-06-11-k7q3mz. Stable, sortable, no spaces, lowercase.
• results push --scoring-report PATH reads run_id from data/manifest.md frontmatter. If the field is missing, it regenerates a new one, prints a warning to stderr, and proceeds. This protects against "I scored against a manifest I forgot to sync" mistakes.
• results push --run-id X is rejected: the value comes from the manifest, not the user. This is the design that prevents the original "hand-typed run id becomes a source of confusion" concern.

Subcommands

• eval-transcript results push --scoring-report PATH [--results-repo REPO] [--message MSG] — append one row per scored (sample_id, provider, model) triple from the scoring report to the configured results dataset, with run_id taken from the manifest.
• eval-transcript results list [--results-repo REPO] [--limit N] — read the results dataset and print a one-line summary per run_id (date, sample count, top WER).

Schema (one row per (run_id, sample_id, provider, model))

column	type	notes
run_id	string	YYYY-MM-DD-xxxxxx
run_started_at	timestamp	ISO 8601 UTC
git_sha	string	short SHA of the working tree at push time; empty if not a git repo
corpus_repo	string	from data/manifest.md (added in issue #30)
corpus_revision	string	from data/manifest.md (added in issue #30)
sample_id	string
provider	string
model	string
wer, cer, mer, wil, wip	float	from the scoring report
hits, substitutions, deletions, insertions, reference_tokens	int	from the scoring report
normalization	string	standard / standard_numbers / raw
transcription_path, ground_truth_path	string	from the scoring report
scorer_version	string	eval-transcript package version (importlib.metadata)

The dataset uses HF Datasets' commit history to append rows. New repos are created with private=True (same hard guard as the corpus dataset, see Constraints).

Auth model

Same as the corpus dataset provider (issue #28): constructor accepts token, defaults to os.getenv("HF_TOKEN"). Writes fail fast without it. Push checks repo_info(...).private and refuses to push to public datasets with no override.

Configuration (.env)

• HF_TOKEN, HF_ORG (optional), HF_RESULTS_REPO (default eval-transcript-results).
• All overridable via CLI flags (--token, --org, --results-repo).

Acceptance criteria

• manifest sync writes a fresh run_id (format YYYY-MM-DD-xxxxxx) on every invocation.
• results push --scoring-report report.json reads run_id from data/manifest.md and appends one row per scored triple to the results dataset.
• results push --run-id X is rejected with a clear error explaining that run ids come from the manifest.
• results push fails fast if HF_TOKEN is missing.
• results push refuses to push to a public dataset repo with a hard error and no override.
• Re-pushing the same (run_id, sample_id, provider, model) row is rejected unless --replace is passed. This protects the ledger from accidental duplicate appends.
• results list reads the results dataset (anonymous when public) and prints one line per run_id (date, sample count, top WER).
• A "Run a benchmark end to end" recipe in the README shows: pull → transcribe → score → push.
• Tests cover: appending a known report, run_id collision, --run-id rejection, token validation, schema round-trip, private-repo guard.
• uv run pre-commit run --all-files is clean.
• uv run python -m unittest discover -s tests -v is clean.

Constraints

• No new runtime dependencies beyond huggingface_hub.
• The --replace flag is the only override; no --allow-public, no --force-corpus-revision-mismatch.
• The run_id generator must be deterministic from (UTC date, random 6-char base32). The same day + the same random suffix cannot be re-generated by the user; that is fine because the generator runs at manifest sync time, not at results push time.
• Scoring report input is the JSON shape produced by eval-transcript score all --format json (see scoring_cli.py render_scores_json).
• Local --output PATH for scoring reports is unchanged; the new push command is additive.

Out of scope

• Comparison/dashboard tools that consume the results dataset.
• Auto-triggering results push from score all (the two commands stay separate; this keeps the local-only path friction-free).
• CI workflow (separate issue).

Depends on

• feat: add Hugging Face dataset pull/push provider (issue feat: add Hugging Face dataset pull/push provider #28).
• feat: add benchmark pull command and dataset-aware manifest sync (issue feat: add benchmark pull command and dataset-aware manifest sync #30) — for corpus_repo / corpus_revision to be available in the manifest.

👾 Generated with Letta Code