Add GitHub Actions core test matrix#23
Conversation
Add a GitHub Actions workflow that installs each harness package in editable mode and runs its stable test_core.py suite. Use a matrix across the 10 reference harnesses so regressions in the pure-Python core modules are caught on pull requests without depending on desktop applications, media fixtures, or API credentials. Allow .github/workflows in the repository ignore rules so workflow files can be tracked. Install lxml explicitly for the Kdenlive and Shotcut jobs because their core tests exercise XML helpers that depend on it.
|
was wondering if such tests are reasonable and meaningful given that E2E and complicated workflow tests actually require the original softwares to be installed |
There was a problem hiding this comment.
Pull request overview
Adds an initial GitHub Actions CI workflow to run each harness package’s stable test_core.py suite, while updating ignore rules so workflow files are tracked.
Changes:
- Add a GitHub Actions workflow with a matrix over the 10 reference harnesses to run
test_core.py. - Install each harness in editable mode with dev dependencies prior to running tests.
- Update
.gitignoreallowlist rules to include.github/workflows/*.yml.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
.gitignore |
Unignore .github/workflows/*.yml so the new workflow can be committed under the repo’s allowlist-style ignore rules. |
.github/workflows/core-tests.yml |
New CI workflow running core test suites per harness via a matrix on Ubuntu + Python 3.11. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
.github/workflows/core-tests.yml
Outdated
| run: | | ||
| python -m pip install -e ".[dev]" | ||
| if [ -n "${{ matrix.extra_pip }}" ]; then | ||
| python -m pip install ${{ matrix.extra_pip }} |
There was a problem hiding this comment.
python -m pip install ${{ matrix.extra_pip }} is unquoted, so if extra_pip ever contains multiple packages or version specifiers with spaces, word-splitting can produce surprising results. Quoting and/or using a YAML list for extra dependencies would make this more robust.
| python -m pip install ${{ matrix.extra_pip }} | |
| python -m pip install "${{ matrix.extra_pip }}" |
| name: Core Tests | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| pull_request: |
There was a problem hiding this comment.
Consider explicitly setting workflow permissions (for example, contents: read) since this job only checks out the repo and runs tests. Relying on the repository default GITHUB_TOKEN permissions is avoidable and can unintentionally grant broader write access than needed.
.github/workflows/core-tests.yml
Outdated
| jobs: | ||
| core-tests: | ||
| name: ${{ matrix.name }} core tests | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
Using runs-on: ubuntu-latest can introduce unexpected CI breakage when GitHub updates the default runner image. Pinning to a specific Ubuntu version (e.g., ubuntu-22.04/24.04) makes the test environment more stable and failures easier to reproduce.
| runs-on: ubuntu-latest | |
| runs-on: ubuntu-22.04 |
Pin the GitHub Actions runner image, declare read-only token permissions, and quote the optional extra dependency install argument to address the automated review suggestions on PR HKUDS#23.
|
@yuh-yang Thanks — I agree these core tests are not a substitute for software-backed E2E or complex workflow coverage. My intent here is to add a first CI layer that is still useful even when the original desktop applications are not installed:
So I see this workflow as baseline regression protection, not as a claim that the harnesses are fully validated end-to-end in CI. A reasonable next step would be to keep this layer and later add a smaller second layer of software-backed smoke/E2E jobs for the harnesses that can be installed headlessly in GitHub Actions. I also pushed a follow-up commit to address the Copilot review suggestions:
|
Summary
test_core.pysuite.github/workflows/*.ymlthrough the repository ignore rules so workflow files can be trackedWhy this scope
The repository currently has no GitHub Actions workflow. The
test_core.pysuites are the highest-signal tests that run reliably without desktop applications, media fixtures, or API credentials, so they are a good first CI baseline.Verification
test_core.pysuite locally:git diff --check