Motivation
Link to the gh-aw PRs: github/gh-aw#57521 (Remove built-in Playwright MCP support, default to CLI mode; mode: mcp now rejected) and github/gh-aw#57594 (Add AWF-aware Playwright CLI policy prompt enforcing loopback-only servers and Squid-routed egress). No existing test exercises tools: playwright in CLI mode at all.
Proposed test
- Workflow file:
test-copilot-playwright-cli.md
- Trigger:
workflow_dispatch
- Engine: copilot
- Safe output:
create-issue (report a short result after running a local Playwright CLI check)
- Variant: standard
Minimal test prompt sketch
Configure tools: { playwright: { mode: cli } } with the sandbox/firewall enabled. Ask the agent to start a local static server bound to 127.0.0.1, use the Playwright CLI to take a screenshot or snapshot of it, then create an issue summarizing success and confirming no external navigation occurred.
New fixtures or secrets needed
None — uses existing repo content for a static page; no new fixtures.
Notes
Verify mode: mcp truly fails to compile (could be asserted with gh aw compile in a lint-only check rather than an E2E dispatch test, since compile-time rejection isn't something workflow_dispatch proves). The CLI-mode happy path is the part suited to E2E; the mode:mcp rejection is more of a compiler unit-test concern.
Generated by 🔍 Suggest New E2E Tests · copilot · auto · 29.7 AIC · ⌖ 4.64 AIC · ⊞ 8.8K · ◷
Motivation
Link to the gh-aw PRs: github/gh-aw#57521 (Remove built-in Playwright MCP support, default to CLI mode;
mode: mcpnow rejected) and github/gh-aw#57594 (Add AWF-aware Playwright CLI policy prompt enforcing loopback-only servers and Squid-routed egress). No existing test exercisestools: playwrightin CLI mode at all.Proposed test
test-copilot-playwright-cli.mdworkflow_dispatchcreate-issue(report a short result after running a local Playwright CLI check)Minimal test prompt sketch
Configure
tools: { playwright: { mode: cli } }with the sandbox/firewall enabled. Ask the agent to start a local static server bound to127.0.0.1, use the Playwright CLI to take a screenshot or snapshot of it, then create an issue summarizing success and confirming no external navigation occurred.New fixtures or secrets needed
None — uses existing repo content for a static page; no new fixtures.
Notes
Verify
mode: mcptruly fails to compile (could be asserted withgh aw compilein a lint-only check rather than an E2E dispatch test, since compile-time rejection isn't somethingworkflow_dispatchproves). The CLI-mode happy path is the part suited to E2E; the mode:mcp rejection is more of a compiler unit-test concern.