Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/patch-playwright-cli-only-guidance.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions .github/aw/playwright.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,12 @@ tools:
version: "0.1.18"
```

`mode: cli` is accepted but unnecessary. `mode: mcp` is not supported by the
built-in tool. If MCP is required, configure and pin `@playwright/mcp` explicitly
under `mcp-servers` and allow only the required tools.
Omit `mode`; the built-in Playwright integration is CLI-only by default. The
explicit `mode: cli` setting remains accepted for compatibility, but it is not
needed and should be removed from workflows that still carry it. `mode: mcp` is
not supported by the built-in tool. If MCP is required, configure and pin
`@playwright/mcp` explicitly under `mcp-servers` and allow only the required
tools.

## Configure network access

Expand Down
1 change: 0 additions & 1 deletion .github/aw/syntax-tools-imports.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,6 @@ The `tools:` field configures which tools the coding agent may use.
```yaml
tools:
playwright:
mode: cli # recommended: token-efficient CLI mode
version: "0.1.11" # optional: @playwright/cli npm package version
```
- `timeout:` - Per-operation timeout in seconds for all tool and MCP calls (integer or expression, default: 60 s for all engines).
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/shared/docs-server-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ This will:

## Playwright Browser Access

With **CLI mode** (`mode: cli`, recommended), `playwright-cli` runs directly on the runner — not in a Docker container. Use `localhost` directly to reach the dev server:
With the built-in Playwright CLI integration, `playwright-cli` runs directly on the runner — not in a Docker container. Use `localhost` directly to reach the dev server:

```bash
playwright-cli browser_navigate --url "http://localhost:4321/gh-aw/"
Expand Down Expand Up @@ -111,7 +111,7 @@ This will:
## Usage Notes

- The server runs on `http://localhost:4321` and is accessible at `http://localhost:4321/gh-aw/` for curl/bash and playwright-cli
- With CLI mode (`mode: cli`), use `localhost` directly for all playwright-cli commands — no bridge IP needed
- With the built-in Playwright CLI integration, use `localhost` directly for all playwright-cli commands — no bridge IP needed
- Always clean up the server when done to avoid orphan processes
- If the server fails to start, check `/tmp/gh-aw/agent/preview.log` for errors
- Node.js >= 22 is required; ensure `runtimes: node: version: "22"` is set in the workflow frontmatter
Expand Down
2 changes: 1 addition & 1 deletion docs/src/content/docs/engines/pi.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ Analyze the repository and create a concise daily status report covering:

## Capabilities and limitations

Pi supports top-level `max-turns`, provider-prefixed models, and `engine.extensions`. Pi already runs in bare mode, so `engine.bare: true` is accepted but has no effect. Pi does not provide native MCP server integration, native `tools.web-search`, per-command bash allowlisting, `max-continuations`, native `engine.agent` selection, or custom `engine.harness` scripts. MCP-backed tools must be exposed through the required CLI proxy.
Pi supports top-level `max-turns`, provider-prefixed models, and `engine.extensions`. Pi already runs in bare mode, so `engine.bare: true` is accepted but has no effect. The built-in `tools.playwright` integration works through `playwright-cli`; omit its `mode` field because CLI is the only built-in mode. Pi does not provide native MCP server integration, native `tools.web-search`, per-command bash allowlisting, `max-continuations`, native `engine.agent` selection, or custom `engine.harness` scripts. MCP-backed tools must be exposed through the required CLI proxy.

See the [AI engine feature comparison](/gh-aw/reference/engines/#engine-feature-comparison) and [Pi extensions reference](/gh-aw/reference/engines/#pi-extensions-extensions).

Expand Down
10 changes: 3 additions & 7 deletions docs/src/content/docs/reference/playwright.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,11 @@ Playwright enables headless browser control for accessibility testing, visual re

## Configuration

The built-in Playwright tool uses CLI mode. It is token-efficient because it does not load MCP tool schemas into the agent context, avoids Docker overhead, and reaches local development servers through `localhost`.
The built-in Playwright tool is CLI-only by default. It is token-efficient because it does not load MCP tool schemas into the agent context, avoids Docker overhead, and reaches local development servers through `localhost`. If an older workflow still sets `mode: cli`, it continues to work for compatibility, but omitting `mode` is preferred.

```yaml wrap
tools:
playwright:
mode: cli
```

The compiler installs `@playwright/cli` as a global npm package, its skills, and
Expand Down Expand Up @@ -52,7 +51,6 @@ The `version` field pins the `@playwright/cli` npm package. Omit it to use the c
```yaml wrap
tools:
playwright:
mode: cli
version: "0.1.18"
```

Expand Down Expand Up @@ -120,7 +118,8 @@ Capture `/tmp/home.png`, then call `upload_artifact` with

## Migrate from Playwright MCP

Remove `mode: mcp` or replace it with `mode: cli`. The compiler now reports `mode: mcp` as an error.
Remove `mode: mcp`. The built-in integration is CLI-only, so no replacement
`mode` field is needed. The compiler now reports `mode: mcp` as an error.
Comment on lines +121 to +122

Replace MCP tool calls in prompts with equivalent `playwright-cli` commands run through bash:

Expand Down Expand Up @@ -173,7 +172,6 @@ on:

tools:
playwright:
mode: cli

network:
allowed:
Expand Down Expand Up @@ -238,7 +236,6 @@ steps:

tools:
playwright:
mode: cli
version: "0.1.18" # pins `@playwright/cli` npm package; see Configuration > Version
bash:
- "npm *"
Expand Down Expand Up @@ -289,7 +286,6 @@ on:

tools:
playwright:
mode: cli
bash: [":*"]

network:
Expand Down
1 change: 0 additions & 1 deletion pkg/workflow/playwright_cli.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ package workflow
//
// tools:
// playwright:
// mode: cli

import (
"strings"
Expand Down
9 changes: 4 additions & 5 deletions pkg/workflow/playwright_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,12 @@
//
// # Migration
//
// To migrate from MCP mode to CLI mode:
// To migrate from MCP mode to the built-in CLI integration:
//
// 1. Add `mode: cli` to your playwright tool configuration:
// 1. Remove `mode: mcp` from your playwright tool configuration:
//
// tools:
// playwright:
// mode: cli
//
// 2. Update prompts to use `playwright-cli <command>` via bash instead of
// MCP browser tool calls. For example:
Expand Down Expand Up @@ -60,15 +59,15 @@ func (c *Compiler) validatePlaywrightMode(workflowData *WorkflowData) error {
"tools.playwright.mode",
mode,
"mode must be a literal value; expressions are not allowed",
"Set mode to cli, or omit mode because CLI is the default",
"Remove mode because the built-in Playwright integration is CLI-only",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guidance is now stricter than the implementation: validatePlaywrightMode still accepts explicit mode: cli, and the repo still contains examples/tests that rely on that accepted form, so telling users to always remove mode is misleading and will create contradictory migration advice.

💡 Align the message with the actual compatibility contract

Right now only mode: mcp and expression-valued mode are rejected. Explicit mode: cli remains valid, which means the validator should either:

  1. keep documenting both accepted forms (mode: cli or omitted), or
  2. start rejecting mode: cli everywhere and update the rest of the tree in the same change.

A message like "Omit mode because CLI is the default, though explicit mode: cli is still accepted for compatibility" would avoid sending users into a dead-end contradiction.

)
}
if mode, ok := config["mode"].(string); ok && strings.EqualFold(mode, "mcp") {
return NewValidationError(
"tools.playwright.mode",
mode,
"built-in Playwright MCP support has been removed",
"Remove `mode: mcp` or change it to `mode: cli`, then update prompts to run `playwright-cli <command>` from bash. If MCP is still required, configure Playwright explicitly under `mcp-servers`. See https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/playwright.md",
"Remove `mode: mcp`, then update prompts to run `playwright-cli <command>` from bash. If MCP is still required, configure Playwright explicitly under `mcp-servers`. See https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/playwright.md",
)
}
if browsers, ok := config["browsers"].([]any); ok {
Expand Down
38 changes: 38 additions & 0 deletions pkg/workflow/playwright_validation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,48 @@ tools:

require.Error(t, err)
assert.Contains(t, err.Error(), "built-in Playwright MCP support has been removed")
assert.Contains(t, err.Error(), "Remove `mode: mcp`")
assert.NotContains(t, err.Error(), "mode: cli")
assert.Contains(t, err.Error(), "playwright-cli <command>")
assert.Contains(t, err.Error(), "mcp-servers")
}

func TestPiEngineAcceptsPlaywrightWithImplicitCLIMode(t *testing.T) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pkg/workflow/playwright_validation_test.go:106: yagni: one-off Pi-engine test for implicit CLI mode. Fold it into the existing TestValidatePiEngineRequirements/TestValidatePlaywrightMode cases to keep coverage local and avoid extra test surface.

tmpDir := t.TempDir()
mdPath := filepath.Join(tmpDir, "test-workflow.md")
content := `---
name: pi-playwright-cli
on: push
engine: pi
permissions:
contents: read
issues: read

tools:
github:
mode: gh-proxy
cli-proxy: true
playwright:
---

# Test Workflow
`
require.NoError(t, os.WriteFile(mdPath, []byte(content), 0o644))

compiler := NewCompiler()
require.NoError(t, compiler.CompileWorkflow(mdPath))

lockPath := filepath.Join(tmpDir, "test-workflow.lock.yml")
lockContent, err := os.ReadFile(lockPath)
require.NoError(t, err)
lockStr := string(lockContent)

assert.Contains(t, lockStr, "@playwright/cli")
assert.Contains(t, lockStr, "playwright-cli install --skills")
assert.NotContains(t, lockStr, "@playwright/mcp")
assert.NotContains(t, lockStr, "mode: cli")
}

// TestCompileWorkflowRejectsLegacyPlaywrightMCPModeWithArgs ensures that a legacy
// configuration combining `mode: mcp` with the removed MCP-only `args` field still
// surfaces the actionable migration error instead of a generic JSON schema
Expand Down
Loading