Problem
The new HF-backed benchmark loop (corpus pull → transcribe → score → results push) should be exercisable from GitHub Actions on a small public corpus so future regressions in the dataset provider, the manifest sync flow, and the results ledger are caught early. Today, the test suite is unit-only and the live HF APIs are never exercised in CI.
Scope
Add a single GitHub Actions workflow, .github/workflows/ci-hf-benchmark.yml, that performs the full loop on a throwaway test org/repo. The workflow runs:
- Install uv + the package.
- Use a repo-level
HF_TOKEN secret bound to a throwaway test org (no real benchmark data).
- Pull two fixed samples from a public test corpus (the same org's
eval-transcript-corpus-test repo).
- Score pre-existing local transcription stubs (no live ASR calls in CI to keep the loop deterministic and free of API costs).
- Push a result row to
eval-transcript-results-test.
- Assert that the result row is queryable after the run.
The workflow uses workflow_dispatch to avoid surprise runs on every PR; it triggers on PRs that touch src/eval_transcript/huggingface.py, src/eval_transcript/results.py, src/eval_transcript/manifest.py, or the workflow file itself.
Acceptance criteria
Constraints
- Do not commit any real
HF_TOKEN to the repo. The workflow reads HF_TOKEN_CI from GitHub secrets.
- The throwaway test org/repo names are documented in the workflow comments; the workflow does not assume they exist (it creates them on first run).
- Live ASR calls are out of scope: the workflow uses pre-existing local transcription stubs in the test org's corpus to keep the loop deterministic and cost-free.
- The workflow does not have write access to
main or any production branch.
Out of scope
- Nightly full-corpus benchmarks.
- Provider comparison dashboards consuming the results dataset.
- Live ASR coverage in CI (still flaky and billed).
Depends on
👾 Generated with Letta Code
Problem
The new HF-backed benchmark loop (corpus pull → transcribe → score → results push) should be exercisable from GitHub Actions on a small public corpus so future regressions in the dataset provider, the manifest sync flow, and the results ledger are caught early. Today, the test suite is unit-only and the live HF APIs are never exercised in CI.
Scope
Add a single GitHub Actions workflow,
.github/workflows/ci-hf-benchmark.yml, that performs the full loop on a throwaway test org/repo. The workflow runs:HF_TOKENsecret bound to a throwaway test org (no real benchmark data).eval-transcript-corpus-testrepo).eval-transcript-results-test.The workflow uses
workflow_dispatchto avoid surprise runs on every PR; it triggers on PRs that touchsrc/eval_transcript/huggingface.py,src/eval_transcript/results.py,src/eval_transcript/manifest.py, or the workflow file itself.Acceptance criteria
.github/workflows/ci-hf-benchmark.ymlexists and parses.workflow_dispatchruns the full loop end to end.uv run pre-commit run --all-filesis clean on the workflow file (.github/**/*.ymlis in scope).Constraints
HF_TOKENto the repo. The workflow readsHF_TOKEN_CIfrom GitHub secrets.mainor any production branch.Out of scope
Depends on
👾 Generated with Letta Code