Skip to content

feat: add benchmark pull command and dataset-aware manifest sync #30

Description

@kaaloo

Problem

The HF dataset provider (issue #28) exposes granular huggingface dataset {ls,pull,push} subcommands, but a typical user wants a single command that fills both data/audio/ and data/ground_truth/ and refreshes data/manifest.md in one go. Today, manifest sync only walks the local filesystem and never records the origin of a sample, so it is impossible to tell which samples came from a remote dataset and at which revision.

This is the aggregator that makes the new HF dataset store ergonomic for a local user, and the building block the upcoming results-ledger issue (results push) needs to know what corpus revision a given run was scored against.

Scope

  • Add eval-transcript benchmark pull that performs huggingface dataset pull + manifest sync in one step.
  • Make manifest sync aware of the HF dataset origin: each sample's rendered frontmatter records source: huggingface and corpus_revision when those are set, with no change to current behavior when the field is absent.
  • The aggregator does not introduce new flags; it just composes the existing huggingface dataset pull and manifest sync flows.

Subcommands

  • eval-transcript benchmark pull [--samples a,b,c] [--all] [--force] [--revision REV] — equivalent to huggingface dataset pull <args> followed by manifest sync. Prints a single summary line: pulled N, refreshed manifest <path>.
  • eval-transcript manifest sync [--corpus-repo REPO] [--corpus-revision REV] [--source huggingface|local] — when --source is huggingface, records source: huggingface and corpus_revision: REV in the rendered manifest frontmatter per sample. Defaults to the existing local-only behavior.

Acceptance criteria

  • benchmark pull --samples a,b is equivalent to huggingface dataset pull --samples a,b followed by manifest sync, and prints a single summary.
  • benchmark pull (no args) pulls the full default sample list from .env (HF_DATASET_REPO).
  • manifest sync records source: huggingface and corpus_revision: REV per sample when those are set; behavior is unchanged when neither flag is passed.
  • Manifest entries rendered from a non-HF source keep the current shape (no source key).
  • Tests use a tmpdir with a fake HF API. The fake returns two sample rows; benchmark pull writes the audio and ground truth files and renders a manifest with source: huggingface per sample.
  • uv run pre-commit run --all-files is clean.
  • uv run python -m unittest discover -s tests -v is clean.

Constraints

  • The aggregator lives in eval_transcript/__init__.py (CLI wiring) and reuses existing helpers in huggingface.py and manifest.py. No new business logic.
  • manifest sync must remain backward compatible: existing manifests with no source key still render correctly.
  • The corpus_revision is recorded verbatim from the value passed to manifest sync. The aggregator pulls the revision from huggingface dataset pull --revision.

Out of scope

  • Results ledger.
  • Auto-pinning the corpus revision in scoring output (the upcoming results-ledger issue will read the manifest's corpus_revision).

Depends on

👾 Generated with Letta Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions