Skip to content

chore: add CI workflow for end-to-end HF pull/score/push #32

Description

@kaaloo

Problem

The new HF-backed benchmark loop (corpus pull → transcribe → score → results push) should be exercisable from GitHub Actions on a small public corpus so future regressions in the dataset provider, the manifest sync flow, and the results ledger are caught early. Today, the test suite is unit-only and the live HF APIs are never exercised in CI.

Scope

Add a single GitHub Actions workflow, .github/workflows/ci-hf-benchmark.yml, that performs the full loop on a throwaway test org/repo. The workflow runs:

  1. Install uv + the package.
  2. Use a repo-level HF_TOKEN secret bound to a throwaway test org (no real benchmark data).
  3. Pull two fixed samples from a public test corpus (the same org's eval-transcript-corpus-test repo).
  4. Score pre-existing local transcription stubs (no live ASR calls in CI to keep the loop deterministic and free of API costs).
  5. Push a result row to eval-transcript-results-test.
  6. Assert that the result row is queryable after the run.

The workflow uses workflow_dispatch to avoid surprise runs on every PR; it triggers on PRs that touch src/eval_transcript/huggingface.py, src/eval_transcript/results.py, src/eval_transcript/manifest.py, or the workflow file itself.

Acceptance criteria

  • Workflow file .github/workflows/ci-hf-benchmark.yml exists and parses.
  • workflow_dispatch runs the full loop end to end.
  • PRs that touch the listed source files trigger the workflow automatically.
  • The workflow fails on any uncaught exception in the round trip (e.g. missing repo, wrong visibility, schema mismatch).
  • A README "Continuous integration" section documents the workflow and the test org convention.
  • uv run pre-commit run --all-files is clean on the workflow file (.github/**/*.yml is in scope).

Constraints

  • Do not commit any real HF_TOKEN to the repo. The workflow reads HF_TOKEN_CI from GitHub secrets.
  • The throwaway test org/repo names are documented in the workflow comments; the workflow does not assume they exist (it creates them on first run).
  • Live ASR calls are out of scope: the workflow uses pre-existing local transcription stubs in the test org's corpus to keep the loop deterministic and cost-free.
  • The workflow does not have write access to main or any production branch.

Out of scope

  • Nightly full-corpus benchmarks.
  • Provider comparison dashboards consuming the results dataset.
  • Live ASR coverage in CI (still flaky and billed).

Depends on

👾 Generated with Letta Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions