Skip to content

Commit 5454948

Browse files
E2E: fixed CI matrix with Playwright sharding; accessibility plan
Replace the spec-folder discovery job with a fixed matrix, as WooCommerce does: a single type-check job, then "Install wizard" (test:e2e:install) and "E2E (i/n)" jobs running test:e2e --shard=i/n. New specs need no workflow change; add a shard entry when the run gets slow. A merged HTML report (blob reporter, upload step and merge-reports job) is included but commented out. todo.md: new §18 accessibility section (axe scans per role, page structure, keyboard and focus, visual adaptability) and §0.7 updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d4e4673 commit 5454948

3 files changed

Lines changed: 140 additions & 42 deletions

File tree

‎.github/workflows/e2e.yml‎

Lines changed: 67 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -13,29 +13,18 @@ concurrency:
1313
cancel-in-progress: true
1414

1515
jobs:
16-
# Finds the spec folders (tests/e2e-pw/tests/<folder> with at least one *.spec.ts) and type-checks the
17-
# suite once, so the matrix below gets one job per folder: install, basic, admin, ...
18-
discover:
19-
name: Find spec folders
16+
typecheck:
17+
name: Type-check the suite
2018
runs-on: ubuntu-latest
2119
timeout-minutes: 10
2220
defaults:
2321
run:
2422
working-directory: tests/e2e-pw
25-
outputs:
26-
suites: ${{ steps.suites.outputs.suites }}
2723

2824
steps:
2925
- name: Checkout
3026
uses: actions/checkout@v4
3127

32-
- name: List spec folders
33-
id: suites
34-
run: |
35-
suites=$(find tests -name '*.spec.ts' | cut -d/ -f2 | sort -u | jq -R . | jq -cs .)
36-
echo "Spec folders: $suites"
37-
echo "suites=$suites" >> "$GITHUB_OUTPUT"
38-
3928
- name: Set up Bun
4029
uses: oven-sh/setup-bun@v2
4130
with:
@@ -44,19 +33,33 @@ jobs:
4433
- name: Install test dependencies
4534
run: bun install --frozen-lockfile
4635

47-
- name: Type-check the suite
36+
- name: Type-check
4837
run: bun run test:e2e:typecheck
4938

39+
# The jobs are listed here rather than discovered from the spec folders. Playwright splits the specs
40+
# between the shards (`--shard=i/n`, by whole spec file since the suite runs serially), so new specs
41+
# need no change here: when the run gets slow, add a shard and bump `n` in every entry.
42+
# Every job has its own stack: the `setup` project installs, seeds and logs every role in first.
43+
# `install` is the exception: its specs need an empty site, so it runs on its own (test:e2e:install
44+
# wipes the stack and runs only the wizard project).
5045
e2e:
51-
name: Playwright (${{ matrix.suite }})
52-
needs: discover
53-
if: needs.discover.outputs.suites != '[]'
46+
name: ${{ matrix.name }}
47+
needs: typecheck
5448
runs-on: ubuntu-latest
5549
timeout-minutes: 45
5650
strategy:
5751
fail-fast: false
5852
matrix:
59-
suite: ${{ fromJSON(needs.discover.outputs.suites) }}
53+
include:
54+
- name: Install wizard
55+
command: test:e2e:install
56+
id: install
57+
- name: E2E (1/2)
58+
command: test:e2e --shard=1/2
59+
id: shard-1
60+
- name: E2E (2/2)
61+
command: test:e2e --shard=2/2
62+
id: shard-2
6063
defaults:
6164
run:
6265
working-directory: tests/e2e-pw
@@ -81,33 +84,64 @@ jobs:
8184
- name: Start the e2e stack
8285
run: bun run e2e:up
8386

84-
# Every job has its own stack: the `setup` project installs, seeds and logs every role in first.
85-
# `install` is the exception: its specs need an empty site, so it runs on its own (test:e2e:install
86-
# wipes the stack and runs only the wizard project).
8787
- name: Run tests
88-
run: |
89-
if [ "${{ matrix.suite }}" = "install" ]; then
90-
bun run test:e2e:install
91-
else
92-
bun run test:e2e "tests/${{ matrix.suite }}"
93-
fi
88+
run: bun run ${{ matrix.command }}
9489
env:
9590
CI: true
9691

9792
# - name: Stack logs
9893
# if: failure()
9994
# run: bun run e2e:compose logs --no-color --timestamps
10095

101-
# - name: Upload report
102-
# if: failure()
96+
# Needs the blob reporter in playwright.config.ts; the `report` job below merges the blobs.
97+
# - name: Upload blob report
98+
# if: ${{ !cancelled() }}
10399
# uses: actions/upload-artifact@v4
104100
# with:
105-
# name: playwright-report-${{ matrix.suite }}
106-
# path: |
107-
# tests/e2e-pw/playwright-report
108-
# tests/e2e-pw/test-results
101+
# name: blob-report-${{ matrix.id }}
102+
# path: tests/e2e-pw/blob-report
109103
# retention-days: 7
110104

111105
- name: Stop the e2e stack
112106
if: always()
113107
run: bun run e2e:compose down -v
108+
109+
# One HTML report for the whole run, merged from every job's blob report.
110+
# report:
111+
# name: Merge reports
112+
# needs: e2e
113+
# if: ${{ !cancelled() }}
114+
# runs-on: ubuntu-latest
115+
# timeout-minutes: 10
116+
# defaults:
117+
# run:
118+
# working-directory: tests/e2e-pw
119+
#
120+
# steps:
121+
# - name: Checkout
122+
# uses: actions/checkout@v4
123+
#
124+
# - name: Set up Bun
125+
# uses: oven-sh/setup-bun@v2
126+
# with:
127+
# bun-version: latest
128+
#
129+
# - name: Install test dependencies
130+
# run: bun install --frozen-lockfile
131+
#
132+
# - name: Download blob reports
133+
# uses: actions/download-artifact@v4
134+
# with:
135+
# pattern: blob-report-*
136+
# merge-multiple: true
137+
# path: tests/e2e-pw/all-blob-reports
138+
#
139+
# - name: Merge into an HTML report
140+
# run: bunx playwright merge-reports --reporter html ./all-blob-reports
141+
#
142+
# - name: Upload HTML report
143+
# uses: actions/upload-artifact@v4
144+
# with:
145+
# name: playwright-report
146+
# path: tests/e2e-pw/playwright-report
147+
# retention-days: 7

‎tests/e2e-pw/playwright.config.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@ export default defineConfig({
3333
reporter: [
3434
['list'],
3535
['html', { outputFolder: './playwright-report', open: 'never' }],
36+
// On CI, each job also writes a blob report that the workflow's `report` job merges into one HTML report.
37+
// ...(process.env.CI ? [['blob', { outputDir: './blob-report' }] as const] : []),
3638
],
3739
use: {
3840
baseURL,

‎tests/e2e-pw/todo.md‎

Lines changed: 71 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -155,18 +155,22 @@ tests/e2e-pw/
155155
├── modules/ # one folder per course module
156156
├── collaboration/ # sessions platform (coordinator/consultant/user)
157157
├── api/ # REST v1, mobile, LTI, OAI-PMH, RSS/iCal feeds
158-
└── i18n/ # el/en smoke
158+
├── i18n/ # el/en smoke
159+
└── a11y/ # axe scans, page structure, keyboard & focus (§18)
159160
```
160161

161162
### 0.7 CI – P1 (done 2026-10-03)
162163
- [x] `.github/workflows/e2e.yml`: on push to `master`, on pull requests and manually; one run per ref (older ones cancelled).
163164
checkout → setup-bun → `bun install --frozen-lockfile` → `playwright install --with-deps chromium` → `test:e2e:typecheck` →
164-
`e2e:up` (image build) → `test:e2e` with `CI=true` → always `down -v`. The "stack logs" and "upload report" steps on failure
165-
are in the file but commented out.
166-
- [x] Matrix per spec folder (2026-10-04): a `discover` job lists `tests/<folder>` with at least one `*.spec.ts` (currently `basic`,
167-
`install`) and runs `test:e2e:typecheck` once; then one `Playwright (<folder>)` job per folder, `fail-fast: false`, each on its
168-
own stack. `install` runs `test:e2e:install` (empty site), the others `test:e2e tests/<folder>` (setup project first). New
169-
folders join the matrix with their first spec. The commented artifact name is per folder (`playwright-report-<folder>`).
165+
`e2e:up` (image build) → `test:e2e` with `CI=true` → always `down -v`. The "stack logs" step on failure is in the file but
166+
commented out.
167+
- [x] ~~Matrix per spec folder (2026-10-04)~~ replaced 2026-10-05 by a fixed matrix with Playwright sharding (as WooCommerce does):
168+
a `typecheck` job runs `test:e2e:typecheck` once; then the `e2e` jobs listed in the workflow, `fail-fast: false`, each on its own
169+
stack: `Install wizard` (`test:e2e:install`, empty site) and `E2E (i/n)` (`test:e2e --shard=i/n`, setup project first in every
170+
shard). Playwright splits by whole spec file (serial suite), so new specs and folders need no workflow change; when the run gets
171+
slow, add a shard entry and bump `n`. Spec files must not depend on another file having run first.
172+
- [ ] Merged report: blob reporter in `playwright.config.ts` (CI only), "upload blob report" step and `report` job
173+
(`playwright merge-reports --reporter html`) are in the files but commented out.
170174
- [x] Checked locally on a copy holding only what git would commit (no host `vendor/`, generated JS or `node_modules`): 6 passed, and
171175
the generated JS is served from the image volumes. Not run on GitHub yet.
172176
- [ ] Optional matrix: `sso` profile on/off, collaboration platform on/off.
@@ -847,16 +851,74 @@ Cross-cutting checks (assert recipients, subject and the link inside):
847851
- [ ] No PHP warnings/notices/`Fatal error` in any page visited by the suite: add an `afterEach` hook that fails on page text matching
848852
`/(Warning|Notice|Deprecated|Fatal error):/` and on uncaught console errors.
849853
- [ ] Mobile viewport project (`devices['Pixel 7']`) for login, portfolio, course home, side menu toggle (P3).
850-
- [ ] P3 – `@axe-core/playwright` scan on homepage, login, portfolio, course home, exercise attempt.
854+
- Accessibility scans and keyboard checks moved to §18.
851855

852856
---
853857

854-
## 18. Suggested order of work
858+
## 18. Accessibility (`tests/a11y`) – P2
859+
Target: WCAG 2.1 AA (the level the Greek public sector must meet under Directive (EU) 2016/2102). Upstream has been doing accessibility
860+
work (e.g. "Group: Accessibility improvements" on `default_mentoring`), so failures found here are worth reporting upstream.
861+
862+
### 18.1 Tooling
863+
- [ ] Add `@axe-core/playwright` to `devDependencies`.
864+
- [ ] `utils/a11y.ts`: `scan(page, { include?, exclude? })` wrapping `AxeBuilder` with tags `wcag2a, wcag2aa, wcag21a, wcag21aa`; attaches the
865+
full JSON result to the test report (`testInfo.attach`) and fails only on violations with impact `serious` or `critical`.
866+
- [ ] Known-violations baseline (`a11y-baseline.json`, keyed by URL pattern + rule id) so the suite can go green on today's markup and
867+
fail only on **new** violations; a `bun run test:e2e:a11y:update` script rewrites it. Shrink the baseline as upstream fixes land.
868+
- [ ] Exclude third-party embeds the platform doesn't control (H5P iframes, video.js, TinyMCE internals, MathJax output) via `exclude`.
869+
- [ ] Add `tests/a11y/` to the 0.6 layout; the CI shards (0.7) pick up its specs automatically.
870+
871+
### 18.2 Automated scans (axe), per role
872+
- [ ] `anon`: homepage, login form, registration (student and teacher request), lost password, course catalog, open course home,
873+
info pages (`info/*.php`, including `accessibility.php` when `activate_accessibility_text` is on), search results.
874+
- [ ] `student`: portfolio, my courses, profile edit, calendar, messages, course home, units, documents, announcements, forum topic,
875+
assignment submit, exercise attempt (one page per question type: unique/multiple answer, fill in blanks, matching, ordering,
876+
drag and drop text/markers, free text, oral), exercise results, questionnaire answer, wiki, glossary.
877+
- [ ] `teacher`: course create wizard, course settings, users list, exercise/question editor, assignment grading, gradebook, attendance,
878+
announcement create (TinyMCE page, editor itself excluded).
879+
- [ ] `admin`: admin index, user search/edit, course search, platform config, theme options.
880+
- [ ] Modals and dynamic states scanned after opening: delete confirmations (bootbox), strict-exam cancellation modal, cloud/upload
881+
dialogs, dropdown menus, accordion panels expanded, form validation errors shown.
882+
- [ ] Each theme in `admin/theme_options.php` that ships with the platform: colour contrast (`color-contrast` rule) on homepage + course home.
883+
884+
### 18.3 Page structure (explicit assertions, not just axe)
885+
- [ ] `<html lang>` matches the interface language (`el`/`en`/…) and switches with the language selector and `course.lang`.
886+
- [ ] Every page has a unique, non-empty `<title>` naming the page (not just the site name).
887+
- [ ] Exactly one `<main>` landmark; header/nav/footer landmarks present; exactly one `<h1>`; heading levels don't skip.
888+
- [ ] Skip link ("Skip to main content") as the first focusable element and it moves focus into `<main>` – none found in
889+
`resources/views` while scanning, so expect this to fail first (report upstream).
890+
- [ ] All form inputs have an accessible name (label, `aria-label` or `aria-labelledby`); required fields expose `required`/`aria-required`;
891+
validation errors are linked with `aria-describedby` and announced (`role="alert"` or live region).
892+
- [ ] Icon-only buttons/links (edit, delete, visibility toggles in module tables) have an accessible name – check with `getByRole(..., { name })`.
893+
- [ ] Images: content images have meaningful `alt`, decorative ones `alt=""`/`aria-hidden`.
894+
- [ ] Data tables (users, gradebook, results) use `<th>` with `scope` or headers; sortable columns announce sort state (`aria-sort`).
895+
896+
### 18.4 Keyboard & focus
897+
- [ ] Log in, open a course, open a module and log out using only `Tab`/`Shift+Tab`/`Enter`/`Space` (no mouse).
898+
- [ ] Focus is always visible (`:focus-visible` outline not removed) – compare screenshots of a focused vs unfocused control on key pages.
899+
- [ ] No keyboard traps: tabbing through the homepage, course home and exercise attempt eventually returns to the address bar/first element.
900+
- [ ] Side menu (course tools) and user dropdown open/close with keyboard; `aria-expanded` toggles accordingly; `Escape` closes them.
901+
- [ ] Modals: focus moves into the modal on open, is trapped inside, `Escape` closes (except the strict-exam modal, which must stay open
902+
until OK), and focus returns to the trigger on close.
903+
- [ ] Exercise question types that rely on drag and drop (ordering, matching, drag and drop text/markers) can be answered by keyboard,
904+
or the gap is recorded as a known issue.
905+
- [ ] Exercise timer / auto-submit warning is announced (live region), not only shown visually.
906+
907+
### 18.5 Visual adaptability – P3
908+
- [ ] 200% zoom (viewport 640×360 CSS px) and 320 px width reflow: no horizontal scroll on homepage, login, course home, exercise attempt.
909+
- [ ] `prefers-reduced-motion: reduce` (`page.emulateMedia`) disables carousel/animations on the homepage.
910+
- [ ] `forced-colors: active` (Windows high contrast emulation): buttons and focus outlines stay visible.
911+
- [ ] Text spacing override (WCAG 1.4.12: line-height 1.5, letter-spacing 0.12em, word-spacing 0.16em) doesn't clip content on key pages.
912+
913+
---
914+
915+
## 19. Suggested order of work
855916
1. §0 infrastructure + §1 seed + `auth.setup.ts` storing sessions for every role.
856917
2. §2 auth and §3 security: first verify the four "found while scanning" items, then §3.1 matrix, §3.3 IDOR, §3.5 CSRF, §3.8 uploads.
857918
3. §6 course lifecycle, §9 exercises, §10 assignments, §11.1/11.2/11.6/11.10/11.12/11.17 (P0 modules).
858919
4. §7 admin users/courses, then config.
859920
5. The remaining modules, §13 APIs, §14 mail, then P2/P3.
921+
6. §18 accessibility: tooling + baseline first, then anon/student scans, then keyboard checks.
860922

861923
*Generated from a scan of `include/init.php` (roles and course access), `include/constants.php`, `modules/*`, `main/*`, `api/v1` and
862924
`resources/views/layouts/partials/sidebarAdmin.blade.php` (admin menu by privilege).*

0 commit comments

Comments
 (0)