Skip to content

Commit 5bab2d9

Browse files
Marve10sMarve10s
authored
feat(testing): E2E runtime, enhanced smoke, and Playwright tests (#103)
* feat(testing): comprehensive testing infrastructure with E2E, smoke, and Playwright Adds three new testing layers for runtime validation of generated projects: **Layer 1: E2E Runtime Tests** (apps/cli/test/e2e/) - Expanded from 3 to 9 test configs covering Next.js, Nuxt, SvelteKit, TanStack Start, React Router, Astro with both fullstack and standalone backends - New startDevServer() runs turbo dev and validates frontend pages load - checkFrontendPage() validates HTML, detects error patterns - checkStaticAssets() verifies CSS/JS bundles load - validateFrameworkPage() checks framework-specific markers **Layer 2: Enhanced Smoke Tests** (testing/) - Added --strict flag: dev-check failures become blocking (not advisory) - 5 new presets: next-payload, astro-sanity, nuxt-fullstack, react-router-hono, tanstack-start-fullstack - Improved HTML validation with framework-specific marker detection - Empty body detection catches render failures **Layer 3: Playwright Web Builder Tests** (apps/web/test/e2e/) - Playwright configured with chromium, auto-starts dev server - data-testid attributes added to stack builder and preview panel - 5 test specs: builder load, CLI command display, URL state, preview file tree, URL sharing restoration **CI Workflow** (.github/workflows/e2e-test.yaml) - Nightly at 4am UTC + manual dispatch - 3 parallel jobs: e2e-runtime, smoke-strict, playwright - Artifact upload for debugging failures Root scripts: test:e2e, test:smoke:strict, test:playwright * feat(testing): add CLI binary tests, interaction tests, typecheck verification, and turbo wiring Addresses gaps in testing coverage: - CLI binary tests: spawn actual `node dist/cli.mjs` with --yes, --yolo, --ecosystem flags, verify project files are created correctly - CLI interaction tests: test flag validation (incompatible combos rejected), ecosystem selection, directory conflict handling via spawned process - TypeScript verification: typecheckProject() runs tsc --noEmit on generated projects to verify zero type errors - Turbo pipeline: test:e2e and test:integration tasks added to turbo.json, `turbo test` now runs all test suites across all packages - Root scripts: test:e2e (CLI), test:e2e:web (Playwright), test:integration Test results: 8/8 interaction tests pass, 7/7 binary tests pass * feat(testing): add compatibility and mobile Playwright test specs Adds the 2 missing Playwright test files: - compatibility.spec.ts: Convex disables options, Payload requires Next.js, Sanity works everywhere - mobile.spec.ts: mobile viewport, no overflow, CLI command visible, landing page Total: 13 Playwright tests across 5 spec files * chore: refresh snapshots and lockfile after rebase onto main
1 parent 773746c commit 5bab2d9

23 files changed

+1392
-204
lines changed

‎.github/workflows/e2e-test.yaml‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
name: E2E & Integration Tests
2+
3+
on:
4+
schedule:
5+
- cron: "0 4 * * *"
6+
workflow_dispatch:
7+
inputs:
8+
run_e2e:
9+
description: "Run CLI E2E runtime tests"
10+
type: boolean
11+
default: true
12+
run_smoke_strict:
13+
description: "Run strict smoke tests (dev-check blocking)"
14+
type: boolean
15+
default: true
16+
run_playwright:
17+
description: "Run Playwright web builder tests"
18+
type: boolean
19+
default: true
20+
21+
permissions:
22+
contents: read
23+
24+
concurrency:
25+
group: e2e-${{ github.ref }}
26+
cancel-in-progress: true
27+
28+
jobs:
29+
e2e-runtime:
30+
name: E2E Runtime Tests
31+
if: ${{ inputs.run_e2e != 'false' }}
32+
runs-on: ubuntu-latest
33+
timeout-minutes: 90
34+
steps:
35+
- uses: actions/checkout@v4
36+
37+
- uses: oven-sh/setup-bun@v2
38+
with:
39+
bun-version: latest
40+
41+
- uses: actions/setup-node@v4
42+
with:
43+
node-version: "22"
44+
45+
- uses: actions/cache@v4
46+
with:
47+
path: |
48+
node_modules
49+
~/.bun/install/cache
50+
key: e2e-${{ runner.os }}-bun-${{ hashFiles('bun.lock') }}
51+
restore-keys: e2e-${{ runner.os }}-bun-
52+
53+
- name: Install Dependencies
54+
run: bun install --frozen-lockfile
55+
56+
- name: Build Packages
57+
run: |
58+
bun run --cwd packages/types build
59+
bun run --cwd packages/template-generator build
60+
bun run --cwd apps/cli build
61+
62+
- name: Run E2E Tests
63+
env:
64+
E2E: "1"
65+
run: bun test apps/cli/test/e2e/e2e.e2e.ts --timeout 600000
66+
67+
- name: Upload E2E Results
68+
if: always()
69+
uses: actions/upload-artifact@v4
70+
with:
71+
name: e2e-results
72+
path: apps/cli/.smoke-e2e/
73+
if-no-files-found: warn
74+
retention-days: 7
75+
76+
smoke-strict:
77+
name: Strict Smoke Tests
78+
if: ${{ inputs.run_smoke_strict != 'false' }}
79+
runs-on: ubuntu-latest
80+
timeout-minutes: 60
81+
steps:
82+
- uses: actions/checkout@v4
83+
84+
- uses: oven-sh/setup-bun@v2
85+
with:
86+
bun-version: latest
87+
88+
- uses: actions/setup-node@v4
89+
with:
90+
node-version: "22"
91+
92+
- uses: actions/cache@v4
93+
with:
94+
path: |
95+
node_modules
96+
~/.bun/install/cache
97+
key: smoke-${{ runner.os }}-bun-${{ hashFiles('bun.lock') }}
98+
restore-keys: smoke-${{ runner.os }}-bun-
99+
100+
- name: Install Dependencies
101+
run: bun install --frozen-lockfile
102+
103+
- name: Build Packages
104+
run: |
105+
bun run --cwd packages/types build
106+
bun run --cwd packages/template-generator build
107+
bun run --cwd apps/cli build
108+
109+
- name: Run Strict Smoke Tests
110+
run: |
111+
bun run testing/smoke-test.ts \
112+
--preset all \
113+
--dev-check \
114+
--strict \
115+
--output testing/.smoke-output
116+
117+
- name: Post Summary
118+
if: always()
119+
run: |
120+
if [ -f testing/.smoke-output/summary.md ]; then
121+
cat testing/.smoke-output/summary.md >> "$GITHUB_STEP_SUMMARY"
122+
fi
123+
124+
- name: Upload Results
125+
if: always()
126+
uses: actions/upload-artifact@v4
127+
with:
128+
name: smoke-strict-results
129+
path: testing/.smoke-output/
130+
if-no-files-found: warn
131+
retention-days: 7
132+
133+
playwright:
134+
name: Playwright Web Builder Tests
135+
if: ${{ inputs.run_playwright != 'false' }}
136+
runs-on: ubuntu-latest
137+
timeout-minutes: 30
138+
steps:
139+
- uses: actions/checkout@v4
140+
141+
- uses: oven-sh/setup-bun@v2
142+
with:
143+
bun-version: latest
144+
145+
- uses: actions/setup-node@v4
146+
with:
147+
node-version: "22"
148+
149+
- uses: actions/cache@v4
150+
with:
151+
path: |
152+
node_modules
153+
~/.bun/install/cache
154+
key: pw-${{ runner.os }}-bun-${{ hashFiles('bun.lock') }}
155+
restore-keys: pw-${{ runner.os }}-bun-
156+
157+
- name: Install Dependencies
158+
run: bun install --frozen-lockfile
159+
160+
- name: Build Packages
161+
run: |
162+
bun run --cwd packages/types build
163+
bun run --cwd packages/template-generator build
164+
165+
- name: Install Playwright Browsers
166+
working-directory: apps/web
167+
run: npx playwright install --with-deps chromium
168+
169+
- name: Run Playwright Tests
170+
working-directory: apps/web
171+
run: npx playwright test
172+
env:
173+
NODE_OPTIONS: "--max-old-space-size=4096"
174+
175+
- name: Upload Report
176+
if: always()
177+
uses: actions/upload-artifact@v4
178+
with:
179+
name: playwright-report
180+
path: apps/web/test-results/
181+
if-no-files-found: warn
182+
retention-days: 7

‎apps/cli/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,8 @@
9999
"test:watch": "bun test --watch",
100100
"test:coverage": "bun test --coverage",
101101
"test:ci": "CI=1 bun test --bail=5",
102-
"test:e2e": "E2E=1 bun test test/e2e/e2e.e2e.ts",
102+
"test:e2e": "E2E=1 bun test test/e2e/e2e.e2e.ts --timeout 600000",
103+
"test:integration": "bun test test/e2e/cli-interaction.test.ts test/e2e/cli-binary.test.ts",
103104
"test:astro-combos": "bun run scripts/test-astro-combinations.ts",
104105
"test:matrix": "MATRIX_MODE=batched bun test ./test/matrix/matrix-test.test.ts",
105106
"test:matrix:fast": "MATRIX_MODE=sample MATRIX_SAMPLE=0.1 bun test ./test/matrix/matrix-test.test.ts",

‎apps/cli/test/__snapshots__/template-snapshots.test.ts.snap‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1177,7 +1177,7 @@ export default defineConfig({
11771177
"db:migrate": "bun run --filter @snapshot-tanstack-router-minimal/db db:migrate",
11781178
"db:local": "bun run --filter @snapshot-tanstack-router-minimal/db db:local"
11791179
},
1180-
"packageManager": "bun@1.3.5",
1180+
"packageManager": "bun@latest",
11811181
"dependencies": {
11821182
"dotenv": "catalog:",
11831183
"zod": "catalog:",
@@ -1785,7 +1785,7 @@ export const trpc = createTRPCOptionsProxy<AppRouter>({
17851785
"db:generate": "bun run --filter @snapshot-next-self-fullstack/db db:generate",
17861786
"db:migrate": "bun run --filter @snapshot-next-self-fullstack/db db:migrate"
17871787
},
1788-
"packageManager": "bun@1.3.5",
1788+
"packageManager": "bun@latest",
17891789
"dependencies": {
17901790
"dotenv": "catalog:",
17911791
"zod": "catalog:",
@@ -2429,7 +2429,7 @@ export const client = createORPCClient<typeof appRouter>(link);
24292429
"db:migrate": "bun run --filter @snapshot-astro-react-integration/db db:migrate",
24302430
"db:local": "bun run --filter @snapshot-astro-react-integration/db db:local"
24312431
},
2432-
"packageManager": "bun@1.3.5",
2432+
"packageManager": "bun@latest",
24332433
"dependencies": {
24342434
"dotenv": "catalog:",
24352435
"zod": "catalog:",
@@ -2939,7 +2939,7 @@ export default defineNuxtConfig({
29392939
"db:migrate": "bun run --filter @snapshot-nuxt-standalone/db db:migrate",
29402940
"db:local": "bun run --filter @snapshot-nuxt-standalone/db db:local"
29412941
},
2942-
"packageManager": "bun@1.3.5",
2942+
"packageManager": "bun@latest",
29432943
"dependencies": {
29442944
"dotenv": "catalog:",
29452945
"zod": "catalog:",
@@ -3698,7 +3698,7 @@ export default defineConfig({
36983698
"db:generate": "bun run --filter @snapshot-express-node-trpc/db db:generate",
36993699
"db:migrate": "bun run --filter @snapshot-express-node-trpc/db db:migrate"
37003700
},
3701-
"packageManager": "bun@1.3.5",
3701+
"packageManager": "bun@latest",
37023702
"dependencies": {
37033703
"dotenv": "catalog:",
37043704
"zod": "catalog:",
@@ -4480,7 +4480,7 @@ export default defineConfig({
44804480
"db:migrate": "bun run --filter @snapshot-hono-bun-orpc/db db:migrate",
44814481
"db:local": "bun run --filter @snapshot-hono-bun-orpc/db db:local"
44824482
},
4483-
"packageManager": "bun@1.3.5",
4483+
"packageManager": "bun@latest",
44844484
"dependencies": {
44854485
"dotenv": "catalog:",
44864486
"zod": "catalog:",
@@ -5276,7 +5276,7 @@ export default defineConfig({
52765276
"db:generate": "bun run --filter @snapshot-better-auth-full/db db:generate",
52775277
"db:migrate": "bun run --filter @snapshot-better-auth-full/db db:migrate"
52785278
},
5279-
"packageManager": "bun@1.3.5",
5279+
"packageManager": "bun@latest",
52805280
"dependencies": {
52815281
"dotenv": "catalog:",
52825282
"zod": "catalog:",
@@ -6101,7 +6101,7 @@ export default defineConfig({
61016101
"dev:server": "bun run --filter @snapshot-convex-clerk/backend dev",
61026102
"dev:setup": "bun run --filter @snapshot-convex-clerk/backend dev:setup"
61036103
},
6104-
"packageManager": "bun@1.3.5",
6104+
"packageManager": "bun@latest",
61056105
"dependencies": {
61066106
"dotenv": "catalog:",
61076107
"zod": "catalog:",
@@ -6594,7 +6594,7 @@ export const trpc = createTRPCOptionsProxy<AppRouter>({
65946594
"db:migrate": "bun run --filter @snapshot-self-next-clerk/db db:migrate",
65956595
"db:local": "bun run --filter @snapshot-self-next-clerk/db db:local"
65966596
},
6597-
"packageManager": "bun@1.3.5",
6597+
"packageManager": "bun@latest",
65986598
"dependencies": {
65996599
"dotenv": "catalog:",
66006600
"zod": "catalog:",
@@ -7275,7 +7275,7 @@ export default defineConfig({
72757275
"db:migrate": "bun run --filter @snapshot-self-tanstack-start-clerk/db db:migrate",
72767276
"db:local": "bun run --filter @snapshot-self-tanstack-start-clerk/db db:local"
72777277
},
7278-
"packageManager": "bun@1.3.5",
7278+
"packageManager": "bun@latest",
72797279
"dependencies": {
72807280
"dotenv": "catalog:",
72817281
"zod": "catalog:",
@@ -8026,7 +8026,7 @@ export default defineConfig({
80268026
"dev:web": "bun run --filter web dev",
80278027
"dev:server": "bun run --filter server dev"
80288028
},
8029-
"packageManager": "bun@1.3.5",
8029+
"packageManager": "bun@latest",
80308030
"dependencies": {
80318031
"dotenv": "catalog:",
80328032
"zod": "catalog:",
@@ -8750,7 +8750,7 @@ export default defineConfig({
87508750
"db:generate": "bun run --filter @snapshot-postgres-prisma/db db:generate",
87518751
"db:migrate": "bun run --filter @snapshot-postgres-prisma/db db:migrate"
87528752
},
8753-
"packageManager": "bun@1.3.5",
8753+
"packageManager": "bun@latest",
87548754
"dependencies": {
87558755
"dotenv": "catalog:",
87568756
"zod": "catalog:",
@@ -9308,7 +9308,7 @@ export default defineConfig({
93089308
"dev:native": "bun run --filter native dev",
93099309
"dev:web": "bun run --filter web dev"
93109310
},
9311-
"packageManager": "bun@1.3.5",
9311+
"packageManager": "bun@latest",
93129312
"dependencies": {
93139313
"dotenv": "catalog:",
93149314
"zod": "catalog:",
@@ -9908,7 +9908,7 @@ export default app;
99089908
"db:migrate": "bun run --filter @snapshot-native-react-native/db db:migrate",
99099909
"db:local": "bun run --filter @snapshot-native-react-native/db db:local"
99109910
},
9911-
"packageManager": "bun@1.3.5",
9911+
"packageManager": "bun@latest",
99129912
"dependencies": {
99139913
"dotenv": "catalog:",
99149914
"zod": "catalog:",

0 commit comments

Comments
 (0)