[E2E][QIT] Migrate Github Actions workflows to work with QIT #11082
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mgascam
changed the base branch from
develop
to
dev/qit-e2e-subscriptions-specs-migration
Test the buildOption 1. Jetpack Beta
Option 2. Jurassic Ninja - available for logged-in A12s🚀 Launch a JN site with this branch 🚀 ℹ️ Install this Tampermonkey script to get more options. Build info:
Note: the build is updated when a new commit is pushed to this PR. |
|
Size Change: 0 B Total Size: 873 kB ℹ️ View Unchanged
|
…ve tag processing logic
…and test matrix for pull requests
…d shopper jobs to avoid duplication
… and shopper jobs, removing exclusion tags to avoid duplication. Remove retries
…led specs, allowing for better error handling and reporting.
…mline test execution paths, enhancing clarity and maintainability.
… to reduce CI load
…improve test directory determination logic for better clarity and maintainability.
…ble retries, enhancing clarity and consistency in test execution.
… improving clarity and flexibility in test execution.
…ne CI configuration
mgascam
force-pushed
the
dev/qit-e2e-enhance-workflows
branch
from
28066f0 to
c763176
Compare
mgascam
changed the title
…-e2e-enhance-workflows
adimoldovan
reviewed
Oct 13, 2025
There was a problem hiding this comment.
Nice to see this moving forward @mgascam!
- Separated Workflows for Different Use Cases:
From what I can see there's not much difference between the 2 workflows, most is duplicated logic (and also quite verbose). I think a single workflow would be better here.
Tag-based filtering: Single tags (e.g., merchant) auto-exclude subscriptions/blocks; multiple tags use AND logic
How much flexibility do you expect you'd need to run these tests manually in GitHub? I think organizing the tests in projects would be easier than dealing with parsing tags in the CI workflow. And use project name(s) as input in the workflow.
| run: | | ||
| RESULTS_JSON="${{ steps.parse_failed_specs.outputs.results_json }}" | ||
|
|
||
| # Extract failed spec file paths from CTRF format |
There was a problem hiding this comment.
Why not using Playwright's last-failed builtin feature? It should greatly simplify the logic, no need to parse any reports.
| required: false | ||
| default: '' | ||
| type: string | ||
| run_ui_mode: |
There was a problem hiding this comment.
What is the benefit of running in UI mode in CI?
|
|
||
| - name: First Run QIT Tests | ||
| id: first_run_qit_tests | ||
| if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} |
There was a problem hiding this comment.
Shouldn't this be == 'pull_request'?
| if [[ "$WORD_COUNT" -eq 1 ]]; then | ||
| # Single tag: exclude subscriptions and blocks tests using --grep-invert with regex alternation | ||
| # Example: --grep @shopper --grep-invert "@(subscriptions|blocks)" | ||
| OPTIONS="${OPTIONS} --pw_test_tag=\"${CLEAN_TAG}\" --pw_options=\"--retries=0 --grep-invert '@(subscriptions|blocks)'\"" |
There was a problem hiding this comment.
I don't understand the need of this step or how it works. What if I use the tag @subscriptions or @blocks?
| # Check if QIT actually ran tests by looking for results directory | ||
| QIT_RESULTS_DIR=$(find /tmp -maxdepth 1 -name "qit-results-*" -type d 2>/dev/null | sort -r | head -1) | ||
| if [[ -z "$QIT_RESULTS_DIR" ]]; then | ||
| echo "::error::QIT did not produce test results. This indicates an environment or setup failure." |
There was a problem hiding this comment.
Would QIT exit with code 0 if the setup failed?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #WOOPMNT-5365
This PR migrates the existing E2E GitHub workflows to use QIT, completing the workflow infrastructure portion of the QIT E2E migration effort. This is part of a stacked PR series where each PR builds incrementally toward full QIT adoption.
Current State & Scope
🚧 Important: This PR focuses on infrastructure migration - moving from legacy E2E workflows to QIT-based workflows. The test suite is functionally complete and running in CI, but stability work is needed as a follow-up task (see WOOPMNT-5361: [E2E][QIT] Fix timeout errors and flaky specs in CI).
The intent is to get the foundation merged and reviewed without blocking on every flaky test. This should allow parallel work on stabilization while infrastructure is in place and track infrastructure changes separately from test fixes.
Changes proposed in this Pull Request
This PR establishes the QIT E2E testing infrastructure with:
qit-e2e-pull-request.yml: Focused PR testing (10 jobs: L-1 + stable WC versions across 5 test categories)qit-e2e.yml(renamed to "Scheduled & Push"): Comprehensive version matrix (25+ jobs including legacy WC 7.7.0, beta, RC, WP nightly).github/scripts/generate-qit-wc-matrix.shwoocommerce-payments.ziponce and uploads as artifactmerchant) auto-exclude subscriptions/blocks; multiple tags use AND logicTesting instructions
What Should Work:
What May Not Work (expected, to be fixed later):
npm run changelogto add a changelog file, choosepatchto leave it empty if the change is not significant. You can add multiple changelog files in one PR by running this command a few times.Post merge