Cdp layer

[Dingway98]

Note

Introduces a full Chrome DevTools Protocol integration for element resolution and interactions, refactors agent flows to use CDP with perf/debug tooling, DOM caching/streaming, updated schemas/docs, and bumps to v1.0.0.

• Core CDP Integration:
  • Add cdp/ layer: frame/session management (FrameContextManager), element resolution, interactions, bounding boxes, Playwright adapter.
  • Expose CDP APIs and debug options; dispose lifecycle on agent close.
• Agent Refactor:
  • Route ai()/aiAction() through CDP (performAction); add runtime context init and CDP fallbacks.
  • Perf logging, step metrics, frame graph snapshots; DOM cache and optional streaming capture.
• A11y DOM Pipeline:
  • Rebuild maps with backendNodeMap, iframe bbox; CDP-based AX/boxes; sync frames/contexts; visual overlay via CDP.
  • Remove legacy DOM injection scripts.
• Actions/Messages:
  • actElement now requires elementId/method/arguments/confidence; shared Playwright/CDP execution.
  • Tweak thinking, wait; update prompts/examples; disable some default actions.
• Extraction & Utilities:
  • Screenshots via CDP; consolidate element finding (captureDOMState).
• Docs:
  • Update README with “CDP First”; add comprehensive docs/cdp-overview.md; remove LangChain migration guide.
• Version/Deps:
  • Bump to 1.0.0; upgrade @anthropic-ai/sdk; minor scripts and .gitignore tweaks.

^{Written by Cursor Bugbot for commit 225b039. This will update automatically on new commits. Configure here.}

[Dingway98]

@claude please review

[cursor]

Bug: Unnecessary DOM Cache Invalidation

The DOM cache is marked dirty immediately after runAction returns, regardless of whether the action succeeded or failed. If an action throws an exception or returns success: false, the cache is still invalidated even though no DOM mutation occurred. This causes unnecessary DOM re-fetches on the next step. The invalidation should only happen when actionOutput.success === true and the action is not read-only.

 

[Dingway98]

@claude review

[Dingway98]

@claude can you review

[claude]

Claude finished @Dingway98's task —— View job

---

Code Review Complete

I've reviewed the CDP layer implementation and related changes. This is an ambitious PR introducing a first-class CDP execution stack. Here's my comprehensive review:

---

Summary

This PR successfully introduces a robust CDP layer infrastructure that provides direct Chrome DevTools Protocol control alongside Playwright. The architecture is well-designed with proper separation of concerns, pooled sessions, and frame context management. However, there are several issues that should be addressed before merging.

---

Critical Issues

1. Listener Overwrite Bug in trackRuntimeForSession (src/cdp/frame-context-manager.ts:663)

Issue: The sessionListeners.set() call at line 663 unconditionally overwrites any existing listeners for a session. When both trackPageEvents and trackRuntimeForSession are called on the same session, the Page event listeners get lost.

// Line 663 - This REPLACES all existing listeners
this.sessionListeners.set(session, [
  { event: "Runtime.executionContextCreated", handler: createdHandler },
  { event: "Runtime.executionContextDestroyed", handler: destroyedHandler },
  { event: "Runtime.executionContextsCleared", handler: clearedHandler },
]);

Impact: Frame lifecycle events stop being tracked when Runtime domain is enabled after Page domain, breaking frame navigation detection.

Fix: Append to existing listeners instead of replacing:

const existing = this.sessionListeners.get(session) ?? [];
this.sessionListeners.set(session, [
  ...existing,
  { event: "Runtime.executionContextCreated", handler: createdHandler },
  // ... other handlers
]);

---

2. Missing backendNodeMap Merge in OOPIF Processing (src/context-providers/a11y-dom/index.ts:747)

Issue: The annotateFrameSessions function merges several maps from OOPIF results but fails to merge backendNodeMap, causing backend node IDs for OOPIF frames to be lost.

Impact: CDP element resolution breaks for cross-origin iframe elements since resolveElement requires backendNodeMap to recover backend node IDs.

Location: Around line 747 in the code that merges sub-tree results from OOPIFs.

---

3. Execution Context Timing Issue (src/cdp/element-resolver.ts:260-272)

Issue: When executionContextId is missing for iframe elements, the code logs a warning but continues with contextId: undefined. This causes XPath evaluation to execute in the wrong context (main frame instead of iframe).

Current Code:

if (frameIndex !== 0 && !executionContextId) {
  if (strict) {
    throw new Error(...);
  }
  console.warn(...); // Just warns, then proceeds with undefined context
}
// Continues to use undefined executionContextId at line 280

Impact: XPath resolution fails or returns wrong elements for iframe content.

Recommendation: Either throw an error in strict mode (already done) or implement a retry/wait mechanism in non-strict mode before proceeding.

---

High Priority Issues

4. Zod Enum Type Mismatch (src/agent/actions/act-element.ts:7)

The automated review flagged this, but it's actually correct. TypeScript's as const satisfies readonly CDPActionMethod[] at line 43 of action-restrictions.ts creates a readonly tuple type, which is compatible with z.enum(). The code is idiomatic and follows best practices.

Status: False positive - no action needed.

---

5. doubleClick Method Missing from Allowed Actions (src/cdp/interactions.ts:19)

Issue: CDPActionMethod type includes doubleClick (line 19) but it's not in AGENT_ELEMENT_ACTIONS or AIACTION_ALLOWED_ACTIONS arrays.

Current State:

• dispatchCDPAction handles doubleClick (line 280-285)
• Action arrays don't include it
• LLM schema validation will reject it

Impact: Low - doubleClick is implemented but not exposed to agents. If this is intentional for now, add a comment explaining why.

Recommendation: Either add "doubleClick" to AGENT_ELEMENT_ACTIONS or remove it from CDPActionMethod type if not ready for production use.

---

Medium Priority Issues

6. Debug Flag Initialization Order (src/agent/index.ts:112)

Issue: setDebugOptions(params.debugOptions, this.debug) is called before this.debug is assigned (line 131).

Current Flow:

Line 112: setDebugOptions(params.debugOptions, this.debug); // this.debug is undefined
// ...
Line 131: this.debug = params.debug ?? false;

Impact: Debug options are initialized with wrong enabled state.

Fix: Move the assignment before the call:

this.debug = params.debug ?? false;
setDebugOptions(params.debugOptions, this.debug);

---

7. Missing Frame Graph Cleanup on Navigation

The DOM cache is properly invalidated on navigation events, but I don't see corresponding cleanup for the frame graph in FrameContextManager. When major navigations occur, stale frame data might persist.

Recommendation: Add a clear() call or selective frame removal when handling major navigation events to prevent memory leaks and stale frame references.

---

Low Priority / Suggestions

8. Performance Optimization Opportunity

In element-resolver.ts, execution context collection has a 500ms timeout. For pages with many iframes, this adds up. Consider:

• Reducing timeout for subsequent frames (first frame 500ms, others 250ms)
• Parallel context collection for multiple frames
• Caching execution contexts more aggressively

---

9. Error Handling Consistency

Some CDP calls use .catch(() => {}) (silent failures) while others properly log warnings. Consider:

• Standardizing error handling patterns
• Adding debug-mode-only logging for silent catches
• Documenting which failures are expected vs. exceptional

---

10. Type Safety in Interactions

dispatchCDPAction uses several as casts for options parsing (lines 278, 290, 293, etc.). Consider using Zod schemas for runtime validation of action arguments to catch mismatches earlier.

---

Positive Observations

1. Excellent Architecture: Clean separation between Playwright adapter, CDP client, session pooling, and frame management
2. Resource Management: Proper session lifecycle with detach handlers and cleanup
3. OOPIF Support: Comprehensive cross-origin iframe handling with parallel discovery
4. Caching Strategy: DOM snapshot cache with proper invalidation on mutations
5. Debug Support: Extensive debug logging and frame graph snapshots
6. Documentation: Well-commented code explaining complex concepts (OOPIF, same-origin iframes, etc.)

---

Testing Recommendations

Before merging, ensure tests cover:

1. Multiple CDP sessions on same page (verify listener preservation)
2. OOPIF element resolution with XPath fallback
3. Frame navigation with context re-initialization
4. Session pool invalidation and recreation
5. Cache invalidation on various navigation events

---

Conclusion

This is a well-engineered foundation for CDP-based automation. The critical issues (listener overwrite, missing backendNodeMap merge, execution context handling) should be fixed before merging. The medium/low priority items can be addressed in follow-up PRs if time-constrained.

Recommendation: Fix critical issues #1, #2, #3 and medium issue #6. Consider #5 (doubleClick) based on product requirements. Then merge.

---

[cursor]

Bug: Incomplete Feature Leaves Dead Code

domChunks is declared as const with value null at line 292 and never reassigned. The conditional at line 355-357 that checks if (domChunks) will always be falsy, making this code unreachable. This appears to be dead code from an incomplete streaming implementation since captureDOMState already handles DOM aggregation internally.