feat(e2e): live-client smoke test harness (Phase 1 + Phase 2) - #45
Draft
jlowapik wants to merge 7 commits into
Draft
feat(e2e): live-client smoke test harness (Phase 1 + Phase 2)#45jlowapik wants to merge 7 commits into
jlowapik wants to merge 7 commits into
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…accounts
Expands the e2e suite from a single read smoke into a per-tool regression
harness that exercises two MCP connectors against two Google accounts:
- awesome-mcp-readonly → mcp-e2e-readonly@ (rich fixtures, never modified)
- awesome-mcp-full → mcp-e2e-write@ (empty; scratch resources only)
Both connectors register through the dashboard against the SAME catalog
slug (google-docs) but each gets a unique instanceId-bound session, which
binds the connector to its own Google OAuth grant. Read-only enforcement
is by manually unchecking write tools on the readonly connector in Claude
Desktop UI (see runbook.md). If that proves fragile, the escalation path
is a google-docs-readonly catalog entry with .readonly OAuth scopes - not
in this PR.
What lands:
- e2e/tools.ts: source-of-truth READ_TOOLS / WRITE_TOOLS arrays
(9 reads, 21 writes derived from src/google-docs/server.ts addTool calls)
- e2e/promptTemplates.ts: preface(mode) helper that names the connector
in each test prompt
- e2e/setup/googleClient.ts: direct Google API client built from a
refresh token (mirrors src/userSession.ts:42-50)
- e2e/setup/scratchFactory.ts: createScratchDoc, createScratchSheet,
trashFile, cleanupScratchFolder - all scoped to e2e-scratch/ folder
- e2e/runSmokeTest.ts: adds mode, setup, teardown, ctx-aware prompts
and assertions
- e2e/tests/read/readGoogleDoc.smoke.ts: updated to use mode:'readonly'
+ preface('readonly'); doubles as fixture-integrity safety net
- e2e/tests/write/appendToGoogleDoc.smoke.ts: first write smoke with
full setup/teardown lifecycle + cleanupScratchFolder safety net in
the test file's after() hook
CI:
- e2e-smoke.yml (deploy gate): now scoped to npm run test:gate which
runs only the 1 read + 1 write smoke. New env wiring for the write
side (E2E_WRITE_GOOGLE_REFRESH_TOKEN, E2E_GOOGLE_CLIENT_ID/SECRET,
E2E_SCRATCH_FOLDER_ID).
- e2e-regression.yml (NEW): nightly cron at 03:00 UTC running the full
tests/**/*.smoke.ts glob. Slack notification on failure (advisory -
does not gate prod).
Docs:
- e2e/fixtures/read.md (renamed from fixtures.md, expanded): readonly
account spec, fixture inventory for Phase 3 to come, GHA variables.
- e2e/fixtures/write.md (new): write account spec, OAuth grant
procedure, scratch folder conventions, GHA secrets.
- e2e/runbook.md: two-connector model, manual write-tool blocking
procedure, account rotation including OAuth and fixture drift.
Phase 2 lands two tests. Phase 3 fills out the remaining 28 tools
as data-driven cases on the same harness.
Verified:
- e2e/ typecheck passes (tsc --noEmit clean)
- Module graph resolves at runtime (test:gate fails at expected
points: Appium not running + fake Google creds)
- Main repo typecheck unchanged
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jlowapik
changed the title
|
Enables driving the e2e runner bootstrap over the tailnet after a one-time console visit. Also ignores .claude/ since its local settings can embed OAuth credentials in permission allowlist entries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 built the read/write dual-connector pattern for google-docs only. Generalize it to every server under src/: google-drive, google-gmail, google-calendar, google-sheets, google-slides, clickup, slack, slack-user. - tools.ts -> tools/<service>.ts + barrel. Lists are derived from each server's `annotations.readOnlyHint`; the derivation reproduces the hand-curated google-docs split exactly, which is what validates it. - promptTemplates: per-service connector pairs, `awesome-mcp-<service>-<mode>`. runSmokeTest now prepends preface(service, mode) itself so a test cannot name the wrong connector. - Per-backend direct-API clients + scratch factories. googleClient gains the slides/gmail/calendar namespaces (same OAuth plumbing); clickup and slack get their own thin clients. Every sweep is double-scoped: container plus the shared `[e2e]` name prefix. - 18 smoke tests (1 read + 1 write per service). - test:gate is an explicit manifest, not a glob. GATE is keyed on WriteTestedService, computed from the WRITE_TOOLS arrays, so a new write-capable service fails typecheck until it is gated. - Wire the new secrets/vars through both e2e workflows. Also fixes a pre-existing bug: `--test "tests/**/*.smoke.ts"` does not glob on Node 20 (workflows pin 20), so the nightly regression matched zero files and reported success. test:full now discovers files explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings PR #45 up to date with main, which had moved ahead by five commits touching the same files the e2e harness owns. Conflict resolutions: - .gitignore — took main's. #45 broadened the rule to a blanket `.claude/`, but main has since checked in `.claude/skills/**` on purpose ("skills are checked in"). Main's narrower rule still ignores settings.local.json, which is the file that actually embeds credentials in allowlist entries, so #45's security intent is preserved without untracking the skills. - .github/workflows/e2e-regression.yml — pinned actions/checkout, setup-node and upload-artifact to the same commit SHAs main uses in e2e-smoke.yml. Not a textual conflict: e2e-regression.yml is new in #45, so it never received 0ff451f's pinning pass and would have landed unpinned against a repo where 7 of 9 workflows are pinned. Carried in from main unchanged: the claude-desktop / chatgpt-web waitForResponseComplete race fixes, persist-credentials: false on checkout, and the Appium --base-path removal (the runbook reference went with it). Verified on the merge result: root typecheck clean, root lint 0 errors, e2e typecheck clean, test:gate and test:full both resolve 18 smoke tests and fail only on unset credentials. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-service tool lists omitted both shared tools for all nine services.
The pattern that generates them matches literal `addTool({name, annotations})`
blocks in src/<service>/server.ts; the shared tools arrive via
registerMintRestBearerForCurl()/registerListRestEndpoints() helpers, so it
never saw them.
That is not cosmetic. mintRestBearerForCurl has readOnlyHint: false, and
runbook.md tells operators to uncheck every tool in a service's WRITE_TOOLS on
its -readonly connector. Absent from every list, it stayed CHECKED on every
readonly connector. The bearer it mints is scoped to the USER, not to a service
and not to reads — createServiceAuth resolves the token to a user and then
looks up that user's connection per route — so a "readonly" connector could
mint a bearer reaching WRITE endpoints on every service the account has
connected. That defeats the entire point of the two-connector split.
mintRestBearerForCurl -> WRITE_TOOLS, listRestEndpoints -> READ_TOOLS, with a
comment in each file recording why they are hand-maintained, and a callout in
runbook.md's blocking step since this is the one checkbox that matters most.
Merging main in also surfaced real drift that had accumulated since these lists
were written, so all nine were regenerated from current source:
- clickup +11 tools (tags, task events, doc images)
- slack-user +7 tools (search, channel events)
- downloadFile was missing from BOTH slack lists: the extraction pattern
required `annotations:` to immediately follow `name:`, and both Slack servers
put an explanatory comment between them. The pattern now tolerates that, and
the addTool-count assertion that caught it is what proved the lists complete.
Counts now: docs 30, drive 15, gmail 14, calendar 6, sheets 12, slides 6,
clickup 45, slack 8, slack-user 15 — each plus the 2 shared.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
jlowapik
added a commit
that referenced
this pull request
Aug 30, 2026
The setup instructions pointed at tests/read/readGoogleDoc.smoke.ts, which does not exist on main — that layout arrives with PR #45. On this branch the single smoke test is tests/readGoogleDoc.smoke.ts, so the command as written would have failed with 'Could not find' on the first thing anyone tried. Also notes that the 18-test figures in the Cost section describe the post-#45 state rather than this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the live-client e2e testing infrastructure in two phases on a single branch. Drives the real Claude Desktop and ChatGPT clients against the dev MCP server from a self-hosted Mac Studio GHA runner.
Phase 1 — scaffold (commits f25ed18, f3aa0eb, 66bcf3c)
e2e/workspace with drivers (Appium-mac2 + Playwright CDP),runSmokeTestdispatch, forensics bundle.readGoogleDoc)..github/workflows/e2e-smoke.ymltriggered onworkflow_runof Deploy → Dev.e2e/mac-studio/install.sh+ launch agent plists).Phase 2 — dual-connector read/write split (commit 6e408f5)
Expands from one read smoke to a per-tool regression harness:
awesome-mcp-readonly+awesome-mcp-full), each bound to its own Google account via separateinstanceId-tagged OAuth grants.e2e/setup/googleClient.tsmirrorssrc/userSession.ts:42-50). Scratch folder isolation;cleanupScratchFolder()safety net.runSmokeTestextended withmode,setup,teardown, ctx-aware prompts and assertions.appendToGoogleDoc.smoke.ts) with full setup → write → verify → teardown lifecycle.e2e/tools.ts: source-of-truthREAD_TOOLS(9) /WRITE_TOOLS(21), derived fromsrc/google-docs/server.tsaddToolcalls. Used by the runbook as the canonical list of tools to disable on the readonly connector..github/workflows/e2e-regression.yml) runs the full glob at 03:00 UTC. Slack notify on failure (advisory — does not gate prod).npm run test:gate= 1 read + 1 write). Phase 3 fills the remaining 28 tools.Deferred (NOT in this PR)
create-tag.ymlprod-gate modification. Flipping the gate before the runner is online blocks prod deploys. The exact diff lives at the bottom ofe2e/runbook.md.e2e/runbook.mdande2e/mac-studio/README.mddocument each step.google-docs-readonlycatalog entry. Documented as the escalation path if manual UI blocking proves fragile.Required operator work before this PR is useful
e2e/mac-studio/install.sh+ manual GHA runner registration).mcp-e2e-readonly@…(pre-populated fixtures) andmcp-e2e-write@…(empty +e2e-scratch/folder).awesome-mcp-readonly+awesome-mcp-full.e2e/tools.tsWRITE_TOOLS).E2E_WRITE_GOOGLE_REFRESH_TOKEN,E2E_GOOGLE_CLIENT_ID,E2E_GOOGLE_CLIENT_SECRET,E2E_SLACK_WEBHOOK_URL.E2E_FIXTURE_DOC_ID,E2E_FIXTURE_DOC_NEEDLE,E2E_SCRATCH_FOLDER_ID.See
e2e/fixtures/read.md,e2e/fixtures/write.md,e2e/runbook.md,e2e/mac-studio/README.md.Verified locally
cd e2e && npm run typecheck— clean.cd e2e && npm run test:gateagainst unset env — fails at the expected points (Appium not running locally, Google API rejects fake creds). Module graph resolves cleanly.npm run typecheckandnpm run lintunchanged.Test plan
[self-hosted, macOS, mac-studio].CLIENT=claude-desktop npm run test:gatefrom the Mac Studio.CLIENT=chatgpt-web.ci → deploy-dev → e2e-smokechain; both gate jobs green; forensics artifacts uploaded.e2e-regression.yml; confirm full glob runs and Slack notifies on injected failure.create-tag.ymldiff per runbook to gate prod tags on Claude Desktop smoke.🤖 Generated with Claude Code