Problem
The HF dataset provider (issue #28) exposes granular huggingface dataset {ls,pull,push} subcommands, but a typical user wants a single command that fills both data/audio/ and data/ground_truth/ and refreshes data/manifest.md in one go. Today, manifest sync only walks the local filesystem and never records the origin of a sample, so it is impossible to tell which samples came from a remote dataset and at which revision.
This is the aggregator that makes the new HF dataset store ergonomic for a local user, and the building block the upcoming results-ledger issue (results push) needs to know what corpus revision a given run was scored against.
Scope
- Add
eval-transcript benchmark pull that performs huggingface dataset pull + manifest sync in one step.
- Make
manifest sync aware of the HF dataset origin: each sample's rendered frontmatter records source: huggingface and corpus_revision when those are set, with no change to current behavior when the field is absent.
- The aggregator does not introduce new flags; it just composes the existing
huggingface dataset pull and manifest sync flows.
Subcommands
eval-transcript benchmark pull [--samples a,b,c] [--all] [--force] [--revision REV] — equivalent to huggingface dataset pull <args> followed by manifest sync. Prints a single summary line: pulled N, refreshed manifest <path>.
eval-transcript manifest sync [--corpus-repo REPO] [--corpus-revision REV] [--source huggingface|local] — when --source is huggingface, records source: huggingface and corpus_revision: REV in the rendered manifest frontmatter per sample. Defaults to the existing local-only behavior.
Acceptance criteria
Constraints
- The aggregator lives in
eval_transcript/__init__.py (CLI wiring) and reuses existing helpers in huggingface.py and manifest.py. No new business logic.
manifest sync must remain backward compatible: existing manifests with no source key still render correctly.
- The
corpus_revision is recorded verbatim from the value passed to manifest sync. The aggregator pulls the revision from huggingface dataset pull --revision.
Out of scope
- Results ledger.
- Auto-pinning the corpus revision in scoring output (the upcoming results-ledger issue will read the manifest's
corpus_revision).
Depends on
👾 Generated with Letta Code
Problem
The HF dataset provider (issue #28) exposes granular
huggingface dataset {ls,pull,push}subcommands, but a typical user wants a single command that fills bothdata/audio/anddata/ground_truth/and refreshesdata/manifest.mdin one go. Today,manifest synconly walks the local filesystem and never records the origin of a sample, so it is impossible to tell which samples came from a remote dataset and at which revision.This is the aggregator that makes the new HF dataset store ergonomic for a local user, and the building block the upcoming results-ledger issue (results push) needs to know what corpus revision a given run was scored against.
Scope
eval-transcript benchmark pullthat performshuggingface dataset pull+manifest syncin one step.manifest syncaware of the HF dataset origin: each sample's rendered frontmatter recordssource: huggingfaceandcorpus_revisionwhen those are set, with no change to current behavior when the field is absent.huggingface dataset pullandmanifest syncflows.Subcommands
eval-transcript benchmark pull [--samples a,b,c] [--all] [--force] [--revision REV]— equivalent tohuggingface dataset pull <args>followed bymanifest sync. Prints a single summary line:pulled N, refreshed manifest <path>.eval-transcript manifest sync [--corpus-repo REPO] [--corpus-revision REV] [--source huggingface|local]— when--sourceishuggingface, recordssource: huggingfaceandcorpus_revision: REVin the rendered manifest frontmatter per sample. Defaults to the existing local-only behavior.Acceptance criteria
benchmark pull --samples a,bis equivalent tohuggingface dataset pull --samples a,bfollowed bymanifest sync, and prints a single summary.benchmark pull(no args) pulls the full default sample list from.env(HF_DATASET_REPO).manifest syncrecordssource: huggingfaceandcorpus_revision: REVper sample when those are set; behavior is unchanged when neither flag is passed.sourcekey).benchmark pullwrites the audio and ground truth files and renders a manifest withsource: huggingfaceper sample.uv run pre-commit run --all-filesis clean.uv run python -m unittest discover -s tests -vis clean.Constraints
eval_transcript/__init__.py(CLI wiring) and reuses existing helpers inhuggingface.pyandmanifest.py. No new business logic.manifest syncmust remain backward compatible: existing manifests with nosourcekey still render correctly.corpus_revisionis recorded verbatim from the value passed tomanifest sync. The aggregator pulls the revision fromhuggingface dataset pull --revision.Out of scope
corpus_revision).Depends on
feat: add Hugging Face dataset pull/push provider(issue feat: add Hugging Face dataset pull/push provider #28).chore: rename source_truth to ground_truth with soft deprecation(issue chore: rename source_truth to ground_truth with soft deprecation #27, transitively for the path names).👾 Generated with Letta Code