Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chrome only #99

Open
wants to merge 17 commits into
base: develop
Choose a base branch
from
254 changes: 127 additions & 127 deletions .github/workflows/e2e_api_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,89 +69,75 @@ jobs:
tests:
name: e2e_api tests
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.49.1-noble
options: --user 1001

steps:
- name: Checkout testing repo
id: clone-dokan-lite
uses: actions/checkout@v4

- name: Use desired version of NodeJS
uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
# - name: Use desired version of NodeJS
# uses: actions/setup-node@v4
# with:
# node-version: 20
# cache: npm
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Restore build steps with Chrome-specific configuration.

Removing Node.js setup and build steps will break the workflow. These steps are essential even for Chrome-only testing.

Restore these steps with Chrome-specific configuration:

+- name: Use desired version of NodeJS
+  uses: actions/setup-node@v4
+  with:
+    node-version: 20
+    cache: npm
+
+- name: Install and build (Chrome-specific)
+  run: |
+    npm ci --ignore-scripts
+    npm run build:chrome

Also applies to: 87-94


- name: Composer install and build (Dokan-lite)
run: |
composer i --no-dev -o || composer update --no-dev -o
# - name: Composer install and build (Dokan-lite)
# run: |
# composer i --no-dev -o || composer update --no-dev -o

- name: Npm install and build (Dokan-lite)
run: |
npm ci || npm i
npm run build
# - name: Npm install and build (Dokan-lite)
# run: |
# npm ci || npm i
# npm run build

# Install test dependencies
- name: Install test dependencies
working-directory: tests/pw
run: |
npm ci || npm i

# # Install test dependencies
# - name: Install browsers
# working-directory: tests/pw
# run: |
# npm run pw:browser-with-deps

# Create wp debuglog file
- name: Create wp debuglog file
id: debug-log
working-directory: tests/pw
run: |
mkdir -p wp-data && touch wp-data/debug.log

# Start wordpress environment
- name: Start WordPress Env
id: wp-env
uses: nick-fields/retry@v3
with:
timeout_minutes: 4
max_attempts: 2
retry_on: error
command: |
cd tests/pw
npm run start:env

# Set permalink structure
- name: Set Permalink structure
working-directory: tests/pw
run: |
npm run wp-env run tests-cli wp rewrite structure /%postname%/
# - name: Create wp debuglog file
# id: debug-log
# working-directory: tests/pw
# run: |
# mkdir -p wp-data && touch wp-data/debug.log

# Activate theme
- name: Activate theme:Storefront
working-directory: tests/pw
run: |
npm run wp-env run tests-cli wp theme activate storefront

# Get Playwright version
- name: Get installed Playwright version
id: playwright-version
working-directory: tests/pw
run: |
echo "PLAYWRIGHT_VERSION=$(npm ls @playwright/test --json | jq --raw-output '.dependencies["@playwright/test"].version')" >> $GITHUB_ENV

# Cache browser binaries, cache key is based on Playwright version and OS
- name: Cache playwright binaries
id: playwright-cache
uses: actions/cache@v4
with:
path: |
~/.cache/ms-playwright
key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
restore-keys: |
${{ runner.os }}-playwright-

# Install browser binaries & OS dependencies if cache missed
- name: Install Playwright browser binaries & OS dependencies
id: pw-install
if: steps.playwright-cache.outputs.cache-hit != 'true'
working-directory: tests/pw
run: |
npm run pw:browser-with-deps
# # Get Playwright version
# - name: Get installed Playwright version
# id: playwright-version
# working-directory: tests/pw
# run: |
# echo "PLAYWRIGHT_VERSION=$(npm ls @playwright/test --json | jq --raw-output '.dependencies["@playwright/test"].version')" >> $GITHUB_ENV

# # Cache browser binaries, cache key is based on Playwright version and OS
# - name: Cache playwright binaries
# id: playwright-cache
# uses: actions/cache@v4
# with:
# path: |
# ~/.cache/ms-playwright
# key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }}
# restore-keys: |
# ${{ runner.os }}-playwright-

# # Install browser binaries & OS dependencies if cache missed
# - name: Install Playwright browser binaries & OS dependencies
# id: pw-install
# if: steps.playwright-cache.outputs.cache-hit != 'true'
# working-directory: tests/pw
# run: |
# npm run pw:browser-with-deps

# # Install only the OS dependencies if cache hit not needed
# - name: Install Playwright OS dependencies
Expand All @@ -160,79 +146,93 @@ jobs:
# run: |
# npm run pw:deps-only

# # Start wordpress environment
# - name: Start WordPress Env
# id: wp-env
# uses: nick-fields/retry@v3
# with:
# timeout_minutes: 4
# max_attempts: 2
# retry_on: error
# command: |
# cd tests/pw
# npm run start:env

Comment on lines +143 to +154
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

WordPress environment setup is required.

The WordPress environment setup is critical for E2E testing, even with Chrome-only changes.

Restore the WordPress setup with proper retry mechanism:

+- name: Start WordPress Env
+  id: wp-env
+  uses: nick-fields/retry@v3
+  with:
+    timeout_minutes: 4
+    max_attempts: 2
+    retry_on: error
+    command: |
+      cd tests/pw
+      npm run start:env:chrome

Committable suggestion skipped: line range outside the PR's diff.


# Run e2e tests
- name: 🧪 Run e2e tests
id: e2e-test
if: success() && (github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'E2E' || github.event.inputs.testsuite == 'All')))
timeout-minutes: 40
working-directory: tests/pw
run: |
npm run test:e2e
npm run test:e2e
Comment on lines +170 to +178
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Improve E2E test execution configuration.

The E2E test execution step lacks proper configuration and error handling.

Enhance the test execution step:

 - name: 🧪 Run e2e tests
   id: e2e-test
   if: success() && (github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'E2E' || github.event.inputs.testsuite == 'All')))
   timeout-minutes: 40
   working-directory: tests/pw
   run: |
-    npm run test:e2e
+    npm run test:e2e:chrome || exit 1
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Run e2e tests
- name: 🧪 Run e2e tests
id: e2e-test
if: success() && (github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'E2E' || github.event.inputs.testsuite == 'All')))
timeout-minutes: 40
working-directory: tests/pw
run: |
npm run test:e2e
npm run test:e2e
# Run e2e tests
- name: 🧪 Run e2e tests
id: e2e-test
if: success() && (github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'E2E' || github.event.inputs.testsuite == 'All')))
timeout-minutes: 40
working-directory: tests/pw
run: |
npm run test:e2e:chrome || exit 1


# Run e2e coverage
- name: 🧪 Run e2e coverage
if: always() && (steps.e2e-test.outcome == 'success' || steps.e2e-test.outcome == 'failure')
working-directory: tests/pw
run: |
npm run test:e2e:coverage

# Run api tests
- name: 🧪 Run api tests
id: api-test
if: always() && steps.wp-env.outcome == 'success' && ( github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'API' || github.event.inputs.testsuite == 'All')))
timeout-minutes: 5
working-directory: tests/pw
run: |
npm run test:api
# # Run e2e coverage
# - name: 🧪 Run e2e coverage
# if: always() && (steps.e2e-test.outcome == 'success' || steps.e2e-test.outcome == 'failure')
# working-directory: tests/pw
# run: |
# npm run test:e2e:coverage

# # Run api tests
# - name: 🧪 Run api tests
# id: api-test
# if: always() && steps.wp-env.outcome == 'success' && ( github.event_name != 'workflow_dispatch' || ( github.event_name == 'workflow_dispatch' && (github.event.inputs.testsuite == 'API' || github.event.inputs.testsuite == 'All')))
# timeout-minutes: 5
# working-directory: tests/pw
# run: |
# npm run test:api

# Prepare test summary
- name: Prepare test summary
id: prepare-test-summary
uses: actions/github-script@v7
if: always() && steps.clone-dokan-lite.outcome == 'success'
with:
result-encoding: string
script: |
const script = require("./tests/pw/utils/gitTestSummary.ts")
return await script({github, context, core})

# # Find PR comment
# - name: Find PR comment by github-actions[bot]
# id: find-comment
# if: github.event_name == 'pull_request'
# uses: peter-evans/find-comment@v3
# with:
# issue-number: ${{ github.event.pull_request.number }}
# comment-author: 'github-actions[bot]'
# body-includes: Tests Summary

# # Post test summary as PR comment
# - name: Create or update PR comment
# if: github.event_name == 'pull_request'
# uses: peter-evans/create-or-update-comment@v4
# - name: Prepare test summary
# id: prepare-test-summary
# uses: actions/github-script@v7
# if: always() && steps.clone-dokan-lite.outcome == 'success'
# with:
# comment-id: ${{ steps.find-comment.outputs.comment-id }}
# issue-number: ${{ github.event.pull_request.number }}
# body: ${{ steps.prepare-test-summary.outputs.result }}
# reactions: hooray
# edit-mode: replace
# result-encoding: string
# script: |
# const script = require("./tests/pw/utils/gitTestSummary.ts")
# return await script({github, context, core})

# # # Find PR comment
# # - name: Find PR comment by github-actions[bot]
# # id: find-comment
# # if: github.event_name == 'pull_request'
# # uses: peter-evans/find-comment@v3
# # with:
# # issue-number: ${{ github.event.pull_request.number }}
# # comment-author: 'github-actions[bot]'
# # body-includes: Tests Summary

# # # Post test summary as PR comment
# # - name: Create or update PR comment
# # if: github.event_name == 'pull_request'
# # uses: peter-evans/create-or-update-comment@v4
# # with:
# # comment-id: ${{ steps.find-comment.outputs.comment-id }}
# # issue-number: ${{ github.event.pull_request.number }}
# # body: ${{ steps.prepare-test-summary.outputs.result }}
# # reactions: hooray
# # edit-mode: replace

# Backup Database
- name: Backup Database
if: always() && steps.wp-env.outcome == 'success'
working-directory: tests/pw
run: |
npm run wp-env run tests-cli wp db export wp-data/db.sql

# Upload artifacts
- name: Archive test artifacts (screenshots, HTML snapshots, Reports)
if: always() && steps.debug-log.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: test-artifact
path: |
tests/pw/wp-data
tests/pw/playwright
tests/pw/playwright-report
if-no-files-found: ignore
retention-days: 30
# - name: Backup Database
# if: always() && steps.wp-env.outcome == 'success'
# working-directory: tests/pw
# run: |
# npm run wp-env run tests-cli wp db export wp-data/db.sql

# # Upload artifacts
# - name: Archive test artifacts (screenshots, HTML snapshots, Reports)
# if: always() && steps.debug-log.outcome == 'success'
# uses: actions/upload-artifact@v4
# with:
# name: test-artifact
# path: |
# tests/pw/wp-data
# tests/pw/playwright
# tests/pw/playwright-report
# if-no-files-found: ignore
# retention-days: 30
2 changes: 1 addition & 1 deletion tests/pw/e2e.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ export default defineConfig({
name: 'e2e_tests',
testMatch: /.*\.spec\.ts/,
/* whether not to run setup tests before running actual tests */
dependencies: NO_SETUP ? [] : ['e2e_setup'],
// dependencies: NO_SETUP ? [] : ['e2e_setup'],
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codebase verification

⚠️ Potential issue

Keep the e2e_setup dependency to maintain test reliability

The e2e_setup dependency is part of a carefully designed test initialization chain:

  • The setup files (_env.setup.ts, _site.setup.ts, etc.) contain crucial environment initialization
  • Other test projects maintain their setup dependencies using NO_SETUP flag
  • Breaking this chain by commenting out the dependency could lead to inconsistent test environments
🔗 Analysis chain

Verify test dependencies before removing setup phase.

Commenting out the e2e_setup dependency could lead to test failures if the environment is not properly initialized. Consider keeping the dependency and using the existing NO_SETUP environment variable instead.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check if any tests depend on environment setup
# Look for setup-dependent operations in test files

# Search for environment-dependent operations
rg -l "beforeAll|beforeEach" tests/e2e/

Length of output: 92


Script:

#!/bin/bash
# First, find test files to understand the directory structure
fd -e ts -e js test

# Then look for setup patterns in the test files
rg -l "beforeAll|beforeEach|setup" tests/pw/

# Check the content of the config file for context
cat tests/pw/e2e.config.ts

Length of output: 12654

/* whether not to run teardown tests after running actual tests */
// teardown: NO_SETUP ? undefined : 'coverage_report',
},
Expand Down
2 changes: 1 addition & 1 deletion tests/pw/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"nid": "npm install --include=dev",
"nip": "npm install --omit=dev",
"pw:chrome-only": "playwright install chromium",
"pw:browser-with-deps": "playwright install --with-deps chromium",
"pw:browser-with-deps": "playwright install --with-deps",
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Align browser installation with Chrome-only objective

The current change installs all browsers, which contradicts the PR's "chrome only" objective and increases CI time unnecessarily.

Apply this fix to install only Chrome:

-        "pw:browser-with-deps": "playwright install --with-deps",
+        "pw:browser-with-deps": "playwright install --with-deps chromium"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"pw:browser-with-deps": "playwright install --with-deps",
"pw:browser-with-deps": "playwright install --with-deps chromium",

"pw:deps-only": "playwright install-deps chromium",
"test": "npx playwright test",
"site:reset": "NO_SETUP=false npm run site:setup && npm run env:setup",
Expand Down
11 changes: 8 additions & 3 deletions tests/pw/tests/e2e/admin.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,14 @@ import { LoginPage } from '@pages/loginPage';
import { data } from '@utils/testData';

test.describe('Admin functionality test', () => {
test('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {
const loginPage = new LoginPage(page);
await loginPage.adminLogin(data.admin);
test.only('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Remove .only to enable all tests.

Using test.only will skip other important tests in the suite, including the logout test. This appears to be a debugging artifact and should be removed before merging.

-    test.only('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {
+    test('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
test.only('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {
test('admin can login', { tag: ['@lite', '@admin'] }, async ({ page }) => {

await page.goto('https://www.google.com/');
await page.getByLabel('Search', { exact: true }).click();
await page.getByLabel('Search', { exact: true }).fill('playwright.dev');
await page.goto(
'https://www.google.com/search?q=playwright.dev&sca_esv=445ea37e19471e67&hl=en&source=hp&ei=p-d8Z4euGMGJ4dUPwLHBuAM&iflsig=AL9hbdgAAAAAZ3z1t0_0F54ioUTUzvVc6jgP2g_yFzvM&ved=0ahUKEwjHnYiumuOKAxXBRLgEHcBYEDcQ4dUDCA4&uact=5&oq=playwright.dev&gs_lp=Egdnd3Mtd2l6Ig5wbGF5d3JpZ2h0LmRldjIFEAAYgAQyBRAAGIAEMgUQABiABDIFEAAYgAQyBRAAGIAEMgUQABiABDIFEAAYgAQyBRAAGIAEMgUQABiABEjbS1CeE1ioR3AGeACQAQCYAbMBoAH8D6oBBDMuMTS4AQPIAQD4AQGYAhegAsoQqAIKwgIKEAAYAxjqAhiPAcICChAuGAMY6gIYjwHCAggQABiABBixA8ICCxAAGIAEGLEDGIMBwgILEC4YgAQYsQMYgwHCAg4QABiABBixAxiDARiKBcICCxAuGIAEGNEDGMcBwgIHEAAYgAQYCsICBBAAGB7CAgYQABgKGB7CAgYQABgIGB7CAggQABgIGAoYHpgDBvEFu5uprVvqxY-SBwQ5LjE0oAeEZg&sclient=gws-wiz',
);
await page.getByRole('link', { name: 'Playwright: Fast and reliable' }).click();
});

test('admin can logout', { tag: ['@lite', '@admin'] }, async ({ page }) => {
Expand Down
Loading