diff --git a/.gitignore b/.gitignore
index a7bd963a..7cb36a85 100644
--- a/.gitignore
+++ b/.gitignore
@@ -7,7 +7,7 @@ yarn-error.log*
lerna-debug.log*
.pnpm-debug.log*
tmp
-debug
+/debug
# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
diff --git a/README.md b/README.md
index 0af20eb2..e6381531 100644
--- a/README.md
+++ b/README.md
@@ -118,7 +118,7 @@ HyperAgent provides two complementary APIs optimized for different use cases:
- β‘ **Fast** - Uses accessibility tree (no screenshots)
- π° **Cheap** - Single LLM call per action
- π― **Reliable** - Direct element finding and execution
-- π **Efficient** - Text-based DOM analysis
+- π **Efficient** - Text-based DOM analysis with automatic ad-frame filtering
**Example**:
```typescript
@@ -136,18 +136,27 @@ await page.aiAction("click the login button");
**Best for**: Complex workflows requiring multiple steps and visual context
**Advantages**:
-- πΌοΈ **Visual Understanding** - Uses screenshots with element overlays
+
+- πΌοΈ **Visual Understanding** - Can use screenshots with element overlays
- π **Complex Tasks** - Handles multi-step workflows automatically
- π§ **Context-Aware** - Better at understanding page layout and relationships
- π **Adaptive** - Can adjust strategy based on page state
+**Parameters**:
+
+- `useDomCache` (boolean): Reuse DOM snapshots for speed
+- `enableVisualMode` (boolean): Enable screenshots and overlays (default: false)
+
**Example**:
+
```typescript
const page = await agent.newPage();
await page.goto("https://flights.google.com");
// Complex task with multiple steps handled automatically
-await page.ai("search for flights from Miami to New Orleans on July 16");
+await page.ai("search for flights from Miami to New Orleans on July 16", {
+ useDomCache: true,
+});
```
### π¨ Mix and Match
@@ -370,6 +379,22 @@ const agent = new HyperAgent({
});
```
+## CDP First
+
+HyperAgent speaks Chrome DevTools Protocol natively. Element lookup, scrolling, typing, frame management, and screenshots all go through CDP so every action has exact coordinates, execution contexts, and browser events. This allows for more custom commands and deep iframe tracking.
+
+HyperAgent integrates seamlessly with Playwright, so you can still use familiar commands, while the actions take full advantage of native CDP protocol with fast locators and advanced iframe tracking.
+
+**Key Features:**
+
+- **Auto-Ad Filtering**: Automatically filters out ad and tracking iframes to keep context clean
+- **Deep Iframe Support**: Tracking across nested and cross-origin iframes (OOPIFs)
+- **Exact Coordinates**: Actions use precise CDP coordinates for reliability
+
+Keep in mind that CDP is still experimental, and stability is not guaranteed. If youβd like the agent to use Playwrightβs native locators/actions instead, set `cdpActions: false` when you create the agent and it will fall back automatically.
+
+The CDP layer is still evolvingβexpect rapid polish (and the occasional sharp edge). If you hit something quirky you can toggle CDP off for that workflow and drop us a bug report.
+
## Contributing
We welcome contributions to Hyperagent! Here's how you can help:
diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md
deleted file mode 100644
index 52e7c8bd..00000000
--- a/docs/MIGRATION.md
+++ /dev/null
@@ -1,203 +0,0 @@
-# LangChain to Native SDK Migration Guide
-
-This guide helps you migrate from LangChain-based HyperAgent to the new native SDK implementation.
-
-## Breaking Changes
-
-### 1. LLM Configuration
-
-**Before (LangChain):**
-```typescript
-import { ChatOpenAI } from "@langchain/openai";
-import { ChatAnthropic } from "@langchain/anthropic";
-
-const agent = new HyperAgent({
- llm: new ChatOpenAI({
- openAIApiKey: process.env.OPENAI_API_KEY,
- modelName: "gpt-4o",
- }),
-});
-```
-
-**After (Native SDK):**
-```typescript
-const agent = new HyperAgent({
- llm: {
- provider: "openai",
- model: "gpt-4o",
- },
-});
-```
-
-### 2. Provider-Specific Configuration
-
-**OpenAI:**
-```typescript
-// Before
-const llm = new ChatOpenAI({
- openAIApiKey: process.env.OPENAI_API_KEY,
- modelName: "gpt-4o",
- temperature: 0.7,
- maxTokens: 1000,
-});
-
-// After
-const agent = new HyperAgent({
- llm: {
- provider: "openai",
- model: "gpt-4o",
- temperature: 0.7,
- maxTokens: 1000,
- },
-});
-```
-
-**Anthropic:**
-```typescript
-// Before
-const llm = new ChatAnthropic({
- anthropicApiKey: process.env.ANTHROPIC_API_KEY,
- modelName: "claude-3-7-sonnet-latest",
-});
-
-// After
-const agent = new HyperAgent({
- llm: {
- provider: "anthropic",
- model: "claude-3-7-sonnet-latest",
- },
-});
-```
-
-**Gemini:**
-```typescript
-// Before (if using LangChain Gemini)
-const llm = new ChatGemini({
- apiKey: process.env.GEMINI_API_KEY,
- modelName: "gemini-2.5-pro-preview-03-25",
-});
-
-// After
-const agent = new HyperAgent({
- llm: {
- provider: "gemini",
- model: "gemini-2.5-pro-preview-03-25",
- },
-});
-```
-
-### 3. Direct LLM Instance Usage
-
-If you were passing a direct LLM instance, you can still do so, but the interface has changed:
-
-**Before:**
-```typescript
-import { ChatOpenAI } from "@langchain/openai";
-
-const llm = new ChatOpenAI({...});
-const agent = new HyperAgent({ llm });
-```
-
-**After:**
-```typescript
-import { createOpenAIClient } from "@hyperbrowser/agent/llm/providers";
-
-const llm = createOpenAIClient({
- apiKey: process.env.OPENAI_API_KEY,
- model: "gpt-4o",
-});
-const agent = new HyperAgent({ llm });
-```
-
-## Migration Steps
-
-### Step 1: Update Dependencies
-
-Remove LangChain dependencies from your `package.json`:
-```bash
-npm uninstall langchain @langchain/core @langchain/openai @langchain/anthropic
-```
-
-The new dependencies are automatically included with HyperAgent.
-
-### Step 2: Update Imports
-
-Remove LangChain imports:
-```typescript
-// Remove these imports
-import { ChatOpenAI } from "@langchain/openai";
-import { ChatAnthropic } from "@langchain/anthropic";
-```
-
-### Step 3: Update Agent Configuration
-
-Replace LangChain LLM instances with configuration objects:
-
-```typescript
-// Before
-const agent = new HyperAgent({
- llm: new ChatOpenAI({
- openAIApiKey: process.env.OPENAI_API_KEY,
- modelName: "gpt-4o",
- }),
-});
-
-// After
-const agent = new HyperAgent({
- llm: {
- provider: "openai",
- model: "gpt-4o",
- },
-});
-```
-
-### Step 4: Update Environment Variables
-
-The environment variable names remain the same:
-- `OPENAI_API_KEY` for OpenAI
-- `ANTHROPIC_API_KEY` for Anthropic
-- `GEMINI_API_KEY` or `GOOGLE_API_KEY` for Gemini
-- `DEEPSEEK_API_KEY` for DeepSeek
-
-### Step 5: Test Your Application
-
-Run your application to ensure everything works correctly:
-
-```bash
-yarn build
-yarn cli -c "Go to hackernews and list the top 3 stories"
-```
-
-## Benefits of Migration
-
-1. **Better Performance**: Native SDKs are optimized for their respective providers
-2. **Reduced Bundle Size**: No LangChain overhead
-3. **Latest Features**: Access to newest provider features immediately
-4. **Better Error Handling**: Provider-specific error messages
-5. **Improved Reliability**: Direct API communication without abstraction layers
-
-## Troubleshooting
-
-### Common Issues
-
-1. **Import Errors**: Make sure to remove all LangChain imports
-2. **Type Errors**: Update your TypeScript types to use the new interfaces
-3. **Configuration Issues**: Double-check your provider and model names
-
-### Getting Help
-
-If you encounter issues during migration:
-
-1. Check the [examples](examples/) directory for working code
-2. Review the [API documentation](docs/)
-3. Open an issue on GitHub
-
-## Rollback Plan
-
-If you need to rollback temporarily:
-
-1. Reinstall LangChain dependencies
-2. Revert your code changes
-3. Use the previous version of HyperAgent
-
-However, we recommend completing the migration for the best experience.
diff --git a/docs/cdp-overview.md b/docs/cdp-overview.md
new file mode 100644
index 00000000..cb27a434
--- /dev/null
+++ b/docs/cdp-overview.md
@@ -0,0 +1,3484 @@
+# CDP / Agent Integration Deep Dive
+
+This document explains every relevant function added or changed since commit `f3cdfb478f5dfc724c24309165ad961c914064e0`. It targets readers with zero context and covers:
+
+1. Frame/session infrastructure and why it exists.
+2. How the accessibility DOM (A11yDOM) pipeline gathers full-frame data.
+3. How bounding boxes/visual overlays work without Playwright.
+4. How `page.aiAction()` (`executeAction`/`runAgentTask`) and `page.ai()` (`executeSingleAction`) flow through CDP.
+5. What each CDP runtime action does under the hood.
+6. **THE BIG PICTURE: Why we have so many maps and events**
+7. Areas that still need abstraction or cleanup.
+
+---
+
+## 0. THE BIG PICTURE: The Map Problem & Why We Need Multiple Event Listeners
+
+### Why So Many Maps?
+
+**TL;DR:** Chrome separates DOM structure, accessibility data, frame information, and execution contexts into different CDP domains. We need multiple maps to stitch them together because Chrome doesn't provide a single unified view.
+
+### The Core Problem: Multiple Identifiers for the Same Element
+
+When you click a button on a webpage with iframes, Chrome needs to track that element using **4 different identifiers**:
+
+1. **`backendNodeId`** (DOM domain) - Identifies the DOM node
+2. **`nodeId`** (Accessibility domain) - Identifies the accessibility tree node
+3. **`frameId`** (Page domain) - Identifies which frame contains the element
+4. **`executionContextId`** (Runtime domain) - Identifies the JavaScript execution context to run scripts in that frame
+
+**The problem:** These IDs are managed by separate CDP domains and are NOT directly linked by Chrome!
+
+### Concrete Example: A Button in an Iframe
+
+Let's say we have this HTML structure:
+
+```html
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+```
+
+When the agent needs to click `#iframe-btn`, here's what we need to resolve:
+
+| **What We Know** | **What We Need to Find** | **Which Map Provides It** |
+|------------------|---------------------------|----------------------------|
+| LLM says: "click the button with text 'I'm in an iframe'" | Which `encodedId` is this element? | `elements` Map (from A11y tree) β **`1-42`** (frame 1, node 42) |
+| `encodedId: "1-42"` | What's the `backendNodeId`? | `backendNodeMap["1-42"]` β **`42`** |
+| `encodedId: "1-42"` | What's the XPath to this element? | `xpathMap["1-42"]` β **`"//button[1]"`** |
+| `frameIndex: 1` | What's the iframe metadata? | `frameMap.get(1)` β `IframeInfo` |
+| `frameIndex: 1` | What's the Chrome `frameId`? | `IframeInfo.frameId` β **`"ABC123"`** OR `FrameContextManager.getFrameIdByIndex(1)` β **`"ABC123"`** |
+| `frameId: "ABC123"` | Which CDP session controls this frame? | `FrameContextManager.getFrameSession("ABC123")` β **`CDPSession`** |
+| `frameId: "ABC123"` | What's the `executionContextId` to run scripts? | `FrameContextManager.getExecutionContextId("ABC123")` β **`5`** |
+| `backendNodeId: 42` + `executionContextId: 5` | How do I click this element? | `resolveElement()` + `dispatchCDPAction()` |
+
+### The 5 Core Maps Explained
+
+#### 1. **`backendNodeMap: Record`**
+**Purpose:** Links our stable `encodedId` to Chrome's DOM `backendNodeId`
+
+**Example Data:**
+```typescript
+{
+ "0-15": 15, // Main frame button
+ "1-42": 42, // Iframe button
+ "1-43": 43, // Iframe input field
+}
+```
+
+**Why?** Chrome DOM APIs require `backendNodeId` to resolve nodes, get bounding boxes, etc. But `backendNodeId` can change on navigation, so we track it.
+
+---
+
+#### 2. **`xpathMap: Record`**
+**Purpose:** Stores XPath to recover elements if `backendNodeId` becomes stale
+
+**Example Data:**
+```typescript
+{
+ "0-15": "//html[1]/body[1]/button[1]",
+ "1-42": "//button[1]", // Relative to iframe document!
+ "1-43": "//input[1]"
+}
+```
+
+**Why?** If a navigation or DOM mutation invalidates our `backendNodeId`, we can re-find the element by evaluating its XPath in the correct execution context.
+
+---
+
+#### 3. **`frameMap: Map`**
+**Purpose:** Tracks metadata about each iframe discovered during DOM traversal
+
+**Example Data (after full pipeline - see note below):**
+```typescript
+Map {
+ 1 => {
+ frameIndex: 1,
+ src: "/child.html",
+ xpath: "//iframe[1]",
+ frameId: "ABC123", // Added later by syncFrameContextManager
+ executionContextId: 5, // Added later by syncFrameContextManager
+ parentFrameIndex: 0,
+ iframeBackendNodeId: 99, // β
From DOM.getDocument
+ contentDocumentBackendNodeId: 100, // β
From DOM.getDocument
+ absoluteBoundingBox: { // Position in main viewport
+ x: 0, y: 200, width: 800, height: 600,
+ top: 200, left: 0, right: 800, bottom: 800
+ }
+ }
+}
+```
+
+**β οΈ CRITICAL: Same-Origin Iframes Don't Have `frameId` Initially**
+
+When `buildBackendIdMaps()` calls `DOM.getDocument({ pierce: true })`, Chrome returns:
+- β
`iframeBackendNodeId` (the ` β
+βββββββββββββββββββββββββββββββββββββββββββ
+
+[frameMap]
+2 β IframeInfo {
+ frameIndex: 2,
+ parentFrameIndex: 1, β Parent is iframe #1
+ frameId: "DEF456",
+ xpath: "//iframe[1]", β Relative to parent frame
+ ...
+}
+
+[backendNodeMap]
+"2-67" β 67
+
+
+OOPIF (Cross-Origin) - frameIndex=3
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+βββββββββββββββββββββββββββββββββββββββββββ
+β β contentDocument NOT in response β
+β (security: cross-origin) β
+β β
+β β
+βββββββββββββββββββββββββββββββββββββββββββ
+
+[frameMap]
+3 β IframeInfo {
+ frameIndex: 3,
+ src: "https://ads.com",
+ frameId: undefined, β Not in DOM response!
+ parentFrameIndex: 0,
+ iframeBackendNodeId: 123,
+ contentDocumentBackendNodeId: undefined, β Can't access
+ executionContextId: undefined
+}
+
+β³ WAIT: Need separate CDP session for OOPIF
+ Will be discovered via captureOOPIFs()
+
+
+After DOM Traversal Complete - Data State Summary:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+MAIN FRAME (index 0):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+From buildBackendIdMaps:
+ [backendNodeMap] "0-15" β 15 β
All elements mapped
+ [xpathMap] "0-15" β "//button[1]" β
All XPaths
+ [tagNameMap] "0-15" β "button" β
All tags
+
+From ensureInitialized (Phase 1):
+ [FrameGraph] { frameId: "ROOT", executionContextId: 5 } β
Ready
+
+Status: β
COMPLETE - Can interact with main frame immediately
+
+
+SAME-ORIGIN IFRAME (index 1):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+From buildBackendIdMaps:
+ [backendNodeMap] "1-42" β 42, "1-43" β 43 β
All elements
+ [xpathMap] "1-42" β "//input[1]" β
All XPaths
+ [frameMap] { iframeBackendNodeId: 99,
+ contentDocBackendNodeId: 100,
+ frameId: undefined } β Missing frameId!
+
+From ensureInitialized (Phase 1):
+ [FrameGraph] { frameId: "ABC123",
+ backendNodeId: 99 } β
Has frameId + backendNodeId
+
+Status: β οΈ NEEDS SYNC - frameMap missing frameId & executionContextId
+ β syncFrameContextManager will match via backendNodeId 99
+
+
+OOPIF IFRAME (index 3):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+From buildBackendIdMaps (main frame):
+ [frameMap] { iframeBackendNodeId: 123,
+ contentDocBackendNodeId: undefined }
+ β NO content (cross-origin blocked!)
+
+From ensureInitialized (Phase 1):
+ [FrameGraph] { frameId: "XYZ789",
+ backendNodeId: 123,
+ sessionId: "oopif-sess" } β
Has separate session
+
+Status: β INCOMPLETE - Need Phase 3 to get OOPIF content
+ β captureOOPIFs will call buildBackendIdMaps(oopifSession)
+
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+NEXT STEPS:
+1. Phase 3: Fetch OOPIF content via their separate CDP sessions
+2. Phase 4: syncFrameContextManager to populate missing frameIds
+3. Phase 5: Fetch accessibility trees for all frames
+```
+
+---
+
+### Phase 3: OOPIF Discovery & Session Creation
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β captureOOPIFs() - For Cross-Origin Frames β
+ββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
+ β
+ ββββββββββββββββββΌβββββββββββββββββ
+ βΌ βΌ βΌ
+[page.frames()] Find frames not Create dedicated
+Playwright API in frameMap CDP session
+ β β β
+ ββββββββββββββββββΌβββββββββββββββββ
+ β
+ βΌ
+ βββββββββββββββββββββββββ
+ β For each OOPIF frame: β
+ βββββββββββββ¬ββββββββββββ
+ β
+ βββββββββββββΌββββββββββββ
+ β newCDPSession() for β
+ β this specific frame β
+ βββββββββββββ¬ββββββββββββ
+ β
+ βββββββββββββΌββββββββββββ
+ β DOM.enable on OOPIF β
+ β session β
+ βββββββββββββ¬ββββββββββββ
+ β
+ βββββββββββββΌββββββββββββ
+ β DOM.getDocument on β
+ β OOPIF session β
+ βββββββββββββ¬ββββββββββββ
+ β
+ βΌ
+ Build maps for OOPIF frame:
+ ββββββββββββββββββββββββββββββ
+ [backendNodeMap]
+ "3-89" β 89 β OOPIF content!
+
+ [frameMap]
+ 3 β IframeInfo {
+ frameIndex: 3,
+ frameId: "GHI789", β Now available!
+ sessionId: "oopif-sess-1",
+ ...
+ }
+
+ [sessions Map]
+ "GHI789" β CDPSession (OOPIF)
+
+Data After OOPIF Discovery:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ββ [backendNodeMap] β
Complete (all frames)
+ββ [xpathMap] β
Complete (all frames)
+ββ [frameMap] β οΈ Still incomplete:
+β β’ frameId: β
Now complete
+β β’ executionContextId: β Still missing!
+ββ Need Phase 4 for execution contexts
+```
+
+---
+
+### Phase 4: Execution Context Collection - Critical Synchronization
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β syncFrameContextManager() - Merge Two Views β
+β (DOM-discovered frames β Event-tracked frames) β
+ββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
+ β
+ ββββββββββββββΌβββββββββββββ
+ β For each IframeInfo in β
+ β frameMap: β
+ ββββββββββββββ¬βββββββββββββ
+ β
+ ββββββββββββββββββββββΌβββββββββββββββββββββ
+ βΌ βΌ βΌ
+Match by frameId Match by Match by
+(if available) backendNodeId Playwright frame
+ β β β
+ ββββββββββββββββββββββΌβββββββββββββββββββββ
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ β MATCHING LOGIC - Concrete Example: β
+ β β
+ β SAME-ORIGIN IFRAME (frameIndex 1): β
+ β βββββββββββββββββββββββββββββββββββββ β
+ β frameMap[1]: β
+ β { iframeBackendNodeId: 99, β Primary matching key! β
+ β frameId: undefined } β
+ β β
+ β FrameContextManager.getFrameByBackendNodeId(99): β
+ β { frameId: "ABC123", backendNodeId: 99 } β Match! β
+ β β
+ β TWO-WAY DATA COPY: β
+ β ββββββββββββββββββββ β
+ β FrameContextManager β frameMap[1]: β
+ β frameMap[1].frameId = "ABC123" β
β
+ β frameMap[1].executionContextId = 5 β
β
+ β β
+ β frameMap[1] β FrameContextManager: β
+ β manager.assignFrameIndex("ABC123", 1) β
β
+ β (Overwrites with DOM traversal order) β
+ βββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββ
+ β
+ βΌ
+ β³ WAIT FOR EXECUTION CONTEXTS
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ Problem: Runtime.executionContextCreated events are async!
+
+ Timeline for each frame:
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ T=0ms Page.frameAttached fires
+ ββ> FrameContextManager.upsertFrame()
+ [frameGraph] has frameId, but no contextId yet
+
+ T=50ms Page.frameNavigated fires
+ ββ> Frame is loading...
+
+ T=120ms Runtime.executionContextCreated fires β Finally!
+ ββ> frameExecutionContexts.set(frameId, contextId)
+
+ T=200ms syncFrameContextManager() runs
+ ββ> Copies contextId into IframeInfo
+
+ β³ If contextId not ready yet, we wait:
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ ββββββββββββββββββββββββββββββββββββββββ
+ β frameManager β
+ β .waitForExecutionContext(frameId, β
+ β timeoutMs: 750) β
+ βββββββββββββ¬βββββββββββββββββββββββββββ
+ β
+ βββββββββΌββββββββ
+ βΌ β βΌ
+ Already Wait Timeout
+ available for after
+ event 750ms
+ β β β
+ βββββββββΌββββββββ
+ β
+ βΌ
+ Returns: contextId | undefined
+
+
+Frame Type-Specific Context Handling:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+Main Frame (index=0):
+ββββββββββββββββββββββββββββββββββββββββ
+Context usually available immediately
+Events tracked on root session
+β
High success rate
+
+Same-Origin Iframe (index=1):
+ββββββββββββββββββββββββββββββββββββββββ
+Shares root session with main frame
+Context created after iframe loads
+β οΈ May need short wait (~100ms)
+
+Nested Same-Origin Iframe (index=2):
+ββββββββββββββββββββββββββββββββββββββββ
+Also on root session
+Context created after parent + child load
+β οΈ May need wait (~200ms)
+
+OOPIF (index=3):
+ββββββββββββββββββββββββββββββββββββββββ
+Separate CDP session
+Context on OOPIF session, not root
+β οΈ Requires captureOOPIFs() first
+β³ May take longer to initialize
+
+
+After Execution Context Collection:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+[frameMap] - NOW COMPLETE!
+1 β IframeInfo {
+ frameIndex: 1,
+ frameId: "ABC123",
+ executionContextId: 5, β
NOW AVAILABLE!
+ sessionId: "root-session",
+ iframeBackendNodeId: 99,
+ ...
+}
+
+[frameExecutionContexts Map]
+"ABC123" β 5
+"DEF456" β 8
+"GHI789" β 12
+
+β
Can now inject scripts and evaluate XPath in correct contexts!
+```
+
+---
+
+### Phase 5: Accessibility Tree Fetch
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β fetchIframeAXTrees() - Get Semantics β
+ββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
+ β
+ ββββββββββββββββββββββΌβββββββββββββββββββββ
+ βΌ βΌ βΌ
+ Main Frame Same-Origin Frames OOPIF Frames
+ β β β
+ βΌ βΌ βΌ
+Accessibility Accessibility Accessibility
+.getFullAXTree() .getPartialAXTree() .getPartialAXTree()
+ (contentDocBackendId) (on OOPIF session)
+ β β β
+ ββββββββββββββββββββββΌβββββββββββββββββββββ
+ β
+ βΌ
+ βββββββββββββββββββββββββββββββββββββββββββββββ
+ Accessibility Nodes Collected Per Frame
+ βββββββββββββββββββββββββββββββββββββββββββββββ
+
+ Frame 0 (main):
+ ββ { nodeId: "ax-1", role: "WebArea", childIds: [...] }
+ ββ { nodeId: "ax-2", role: "button", name: "Login",
+ β backendDOMNodeId: 15 } β LINKS TO DOM!
+ ββ ...
+
+ Frame 1 (same-origin iframe):
+ ββ { nodeId: "ax-10", role: "textbox", name: "Email",
+ β backendDOMNodeId: 42 } β LINKS TO DOM!
+ ββ ...
+
+ Frame 3 (OOPIF):
+ ββ { nodeId: "ax-20", role: "button", name: "Ad Click",
+ β backendDOMNodeId: 89 } β LINKS TO DOM!
+ ββ ...
+
+Key Connection:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+backendDOMNodeId (from AX tree) === backendNodeId (from DOM tree)
+
+This is how we merge semantic data with structural data!
+```
+
+---
+
+### Phase 6: Build Hierarchical Tree - Merge All Data
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β buildHierarchicalTree() - Final Assembly β
+ββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
+ β
+ ββββββββββββββββββΌβββββββββββββββββ
+ β For each frame's AX nodes: β
+ ββββββββββββββββββ¬βββββββββββββββββ
+ β
+ ββββββββββββββββββΌβββββββββββββββββ
+ β For each AX node with β
+ β backendDOMNodeId: β
+ ββββββββββββββββββ¬βββββββββββββββββ
+ β
+ βΌ
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ CREATE ENRICHED ELEMENT
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ Input from different sources:
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ [AX Node] [DOM Maps]
+ role: "button" tagNameMap["1-42"] = "button"
+ name: "Login" xpathMap["1-42"] = "//button[1]"
+ backendDOMNodeId: 42 backendNodeMap["1-42"] = 42
+ frameMap.get(1) = IframeInfo {...}
+
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββ
+ β encodedId = createEncodedId( β
+ β frameIndex, β
+ β backendDOMNodeId β
+ β ) β
+ β // "1-42" β
+ ββββββββββββββββββ¬ββββββββββββββββββββ
+ β
+ βΌ
+ [AccessibilityNode with encodedId]
+ {
+ encodedId: "1-42", β Universal key!
+ role: "button",
+ name: "Login",
+ backendDOMNodeId: 42,
+ xpath: "//button[1]", β From xpathMap
+ frameIndex: 1, β Embedded in encodedId
+ // Can now resolve to:
+ // - CDP session (via frameMap)
+ // - executionContextId (via frameMap)
+ // - XPath (via xpathMap)
+ }
+
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββ
+ β elements.set(encodedId, node) β
+ ββββββββββββββββββββββββββββββββββββββ
+
+
+Optional: Bounding Boxes (Visual Mode Only)
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+If mode === "visual-debug":
+
+ββββββββββββββββββββββββββββββββββββββββββ
+β batchCollectBoundingBoxes β
+β Via CDP for each frame β
+βββββββββββββββββ¬βββββββββββββββββββββββββ
+ β
+ β³ REQUIRES: executionContextId per frame
+ β
+ βΌ
+ For each frame with executionContextId:
+ ββββββββββββββββββββββββββββββββββββββββ
+ 1. ensureScriptInjected(session, contextId)
+ ββ> Inject window.__hyperagent_collectBoundingBoxesByXPath
+
+ 2. Runtime.evaluate with contextId
+ ββ> Call injected function with XPath map
+
+ 3. Get coordinates for each element
+
+ 4. If iframe: translate coordinates
+ ββ> Add iframe.absoluteBoundingBox offset
+
+ β
+ βΌ
+ [boundingBoxMap]
+ "0-15" β { x: 100, y: 200, width: 80, height: 40, ... }
+ "1-42" β { x: 50, y: 500, width: 100, height: 35, ... }
+ ββ Already translated to main viewport!
+```
+
+---
+
+### Phase 7: Final State Assembly - Ready for LLM
+
+```
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β ALL DATA COLLECTED β
β
+ββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββ
+ β
+ ββββββββββββββββββΌβββββββββββββββββ
+ β Assemble A11yDOMState β
+ ββββββββββββββββββ¬βββββββββββββββββ
+ β
+ βΌ
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ COMPLETE DATA STRUCTURE
+ βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+ A11yDOMState {
+
+ // For LLM:
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ simplified: string
+ "[0-15] button 'Login'
+ [0-23] textbox 'Email'
+
+ Frame 1 (/same-origin.html):
+ [1-42] button 'Submit'
+ [1-43] textbox 'Password'
+
+ Frame 3 (https://ads.com):
+ [3-89] link 'Click Here'"
+
+ elements: Map
+ "0-15" β { role, name, encodedId, ... }
+ "1-42" β { role, name, encodedId, ... }
+ "3-89" β { role, name, encodedId, ... }
+
+ screenshot?: Buffer (if visual mode)
+ overlayImage?: Buffer (if visual mode)
+
+ // For Element Resolution:
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ backendNodeMap: Record
+ "0-15" β 15
+ "1-42" β 42
+ "3-89" β 89
+
+ xpathMap: Record
+ "0-15" β "//button[1]"
+ "1-42" β "//button[1]" (relative to frame)
+ "3-89" β "//a[1]" (in OOPIF)
+
+ frameMap: Map
+ 0 β { frameIndex: 0, frameId: "ROOT", ... }
+ 1 β { frameIndex: 1, frameId: "ABC123",
+ executionContextId: 5, ... } β
COMPLETE
+ 3 β { frameIndex: 3, frameId: "GHI789",
+ executionContextId: 12, β
COMPLETE
+ sessionId: "oopif-sess-1", ... }
+
+ boundingBoxMap?: Map
+ "0-15" β { x: 100, y: 200, ... }
+ "1-42" β { x: 50, y: 500, ... } (translated!)
+
+ // Metadata:
+ ββββββββββββββββββββββββββββββββββββββββββββββββββββ
+ domState: string (raw tree)
+ metrics: {
+ totalElements: 156,
+ frameCount: 3,
+ captureTimeMs: 450
+ }
+ }
+
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββ
+ β Return to caller β
+ β (executeSingleAction or β
+ β runAgentTask) β
+ ββββββββββββββββββ¬ββββββββββββββββββββ
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββ
+ β LLM Call with DOM State β
+ β β
+ β LLM receives: β
+ β β’ simplified tree β
+ β β’ screenshot (if visual mode) β
+ β β
+ β LLM returns: β
+ β { elementId: "1-42", β
+ β method: "click", β
+ β arguments: [] } β
+ ββββββββββββββββββ¬ββββββββββββββββββββ
+ β
+ βΌ
+ ββββββββββββββββββββββββββββββββββββββ
+ β resolveElement("1-42", { β
+ β frameMap, β
+ β backendNodeMap, β
+ β xpathMap, β
+ β frameContextManager β
+ β }) β
+ β β
+ β Uses ALL the data we collected! β
β
+ ββββββββββββββββββββββββββββββββββββββ
+```
+
+---
+
+### Summary: Critical Synchronization Points
+
+```
+DATA COLLECTION STAGES:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+Stage 1: Event Listeners
+ββ FrameContextManager initialized
+ββ Listening for Page/Runtime events
+ββ Data: [FrameGraph] skeleton, [sessions], [frameExecutionContexts] empty
+
+Stage 2: DOM Traversal
+ββ buildBackendIdMaps() via DOM.getDocument
+ββ Same-origin frames: β
Complete DOM data
+ββ OOPIF frames: β οΈ Only iframe element, no content
+ββ Data: [backendNodeMap], [xpathMap], [frameMap] (partial)
+
+β³ SYNC POINT 1: OOPIF Discovery
+ββ captureOOPIFs() for cross-origin frames
+ββ Create dedicated CDP sessions
+ββ Data: [frameMap] updated with OOPIF frameIds
+
+Stage 3: Execution Context Wait
+ββ syncFrameContextManager() merges views
+ββ waitForExecutionContext() for each frame
+ββ β³ CRITICAL: Block until contexts available
+
+β³ SYNC POINT 2: Execution Contexts Ready
+ββ All frames have executionContextId
+ββ Can now inject scripts
+ββ Data: [frameMap] complete with contextIds
+
+Stage 4: Accessibility Tree
+ββ fetchIframeAXTrees() via Accessibility domain
+ββ Data: AX nodes with backendDOMNodeId linkage
+
+Stage 5: Merge & Enrich
+ββ buildHierarchicalTree() creates encodedIds
+ββ Optional: batchCollectBoundingBoxes()
+ββ Data: [elements] Map, [boundingBoxMap]
+
+Stage 6: Assembly
+ββ Create final A11yDOMState
+ββ β
READY FOR LLM
+
+
+FRAME TYPE TIMINGS (Typical):
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+Main Frame: ~50ms (immediate)
+Same-Origin Iframe: ~150ms (DOM + context wait)
+Nested Same-Origin: ~250ms (parent + child loading)
+OOPIF: ~400ms (session creation + DOM + context)
+
+Total for complex page with 2 same-origin + 1 OOPIF:
+~450-600ms before LLM call can be made
+
+
+FINAL DATA STATE - BY FRAME TYPE:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+MAIN FRAME (frameIndex 0):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+{
+ // From buildBackendIdMaps:
+ backendNodeMap: { "0-15": 15, "0-224": 224, ... },
+ xpathMap: { "0-15": "//button[1]", ... },
+
+ // From ensureInitialized + syncFrameContextManager:
+ frameMap: Map {
+ 0 => {
+ frameIndex: 0,
+ frameId: "ROOT_123",
+ executionContextId: 5, β
Available immediately
+ sessionId: "root-session",
+ url: "https://example.com"
+ }
+ },
+
+ // From buildHierarchicalTree:
+ elements: Map {
+ "0-15" => {
+ encodedId: "0-15",
+ role: "button",
+ name: "Login",
+ backendDOMNodeId: 15,
+ xpath: "//button[1]",
+ frameIndex: 0
+ }
+ }
+}
+
+SAME-ORIGIN IFRAME (frameIndex 1):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+{
+ // From buildBackendIdMaps (pierce: true captured content):
+ backendNodeMap: { "1-42": 42, "1-43": 43, ... },
+ xpathMap: { "1-42": "//input[1]", ... }, β Relative to iframe
+
+ // From ensureInitialized β DOM.getFrameOwner:
+ // FrameGraph had: { frameId: "ABC123", backendNodeId: 99 }
+
+ // From syncFrameContextManager (matched by backendNodeId 99):
+ frameMap: Map {
+ 1 => {
+ frameIndex: 1,
+ frameId: "ABC123", β Matched via backendNodeId!
+ executionContextId: 5, β β
From events
+ sessionId: "root-session", β Shares main session
+ parentFrameIndex: 0,
+ iframeBackendNodeId: 99,
+ contentDocBackendNodeId: 100,
+ src: "/child.html",
+ absoluteBoundingBox: { ... }
+ }
+ },
+
+ // From buildHierarchicalTree:
+ elements: Map {
+ "1-42" => {
+ encodedId: "1-42",
+ role: "textbox",
+ name: "Email",
+ backendDOMNodeId: 42,
+ xpath: "//input[1]", β Relative to frame 1
+ frameIndex: 1
+ }
+ }
+}
+
+OOPIF / CROSS-ORIGIN IFRAME (frameIndex 3):
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+{
+ // From buildBackendIdMaps (main frame):
+ // β OOPIF content NOT captured (cross-origin blocked)
+
+ // From captureOOPIFs β buildBackendIdMaps(oopifSession, pierce: false):
+ backendNodeMap: { "3-89": 89, "3-90": 90, ... },
+ xpathMap: { "3-89": "//a[1]", ... }, β From OOPIF session
+
+ // From ensureInitialized β captureOOPIFs:
+ // Created separate CDP session via context.newCDPSession()
+ // FrameGraph had: { frameId: "XYZ789", backendNodeId: 123, sessionId: "oopif-1" }
+
+ // From syncFrameContextManager:
+ frameMap: Map {
+ 3 => {
+ frameIndex: 3,
+ frameId: "XYZ789", β From captureOOPIFs
+ executionContextId: 12, β β
From OOPIF session events
+ sessionId: "oopif-session-1", β β
Separate session!
+ parentFrameIndex: 0,
+ iframeBackendNodeId: 123, β From main frame DOM
+ contentDocBackendNodeId: undefined, β Not accessible from main
+ src: "https://ads.com/banner",
+ absoluteBoundingBox: { ... }
+ }
+ },
+
+ // From buildHierarchicalTree:
+ elements: Map {
+ "3-89" => {
+ encodedId: "3-89",
+ role: "link",
+ name: "Click Ad",
+ backendDOMNodeId: 89, β From OOPIF's DOM
+ xpath: "//a[1]", β Relative to frame 3
+ frameIndex: 3
+ }
+ }
+}
+
+KEY DIFFERENCES:
+ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+β’ Main Frame: executionContextId immediate, no parent
+β’ Same-Origin: Matched via backendNodeId, shares root session
+β’ OOPIF: Separate session, discovered via Target events, pierce:false
+
+
+WHY WE NEED ALL THIS DATA:
+βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
+
+LLM returns: "1-42"
+
+To act on it, we need:
+ββ backendNodeMap["1-42"] β 42 (which DOM node)
+ββ xpathMap["1-42"] β "//button[1]" (fallback if stale)
+ββ frameMap.get(1) β IframeInfo { (which frame)
+β frameId: "ABC123", (for session lookup)
+β executionContextId: 5, (for XPath evaluation)
+β sessionId: "root" or "oopif-..." (which CDP connection)
+β }
+ββ frameContextManager.getFrameSession("ABC123") β CDPSession
+
+Without ANY of these pieces, element resolution fails! π«
+```
+
+This flow diagram shows why the synchronization is so complex and why we need multiple maps and event listeners working together.
+
diff --git a/examples/output-to-schema/output-to-schema.ts b/examples/output-to-schema/output-to-schema.ts
index a19979c7..55b2c8c8 100644
--- a/examples/output-to-schema/output-to-schema.ts
+++ b/examples/output-to-schema/output-to-schema.ts
@@ -40,7 +40,18 @@ async function runEval() {
provider: "openai",
model: "gpt-4o",
},
- debug: true,
+ // llm: {
+ // provider: "anthropic",
+ // model: "claude-sonnet-4-0",
+ // },
+ // debug: true,
+ browserProvider: "Hyperbrowser",
+ cdpActions: true,
+ debugOptions: {
+ cdpSessions: true,
+ traceWait: true,
+ profileDomCapture: true,
+ },
});
await sleep(1000);
@@ -60,6 +71,8 @@ async function runEval() {
releaseYear: z.number().describe("The year the movie was released"),
rating: z.string().describe("The IMDb rating of the movie"),
}),
+ useDomCache: true,
+ // enableVisualMode: true,
});
await agent.closeAgent();
console.log(chalk.green.bold("\nResult:"));
diff --git a/package.json b/package.json
index c789e485..d9737839 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "@hyperbrowser/agent",
- "version": "0.12.0",
+ "version": "1.0.0",
"description": "Hyperbrowsers Web Agent",
"author": "",
"main": "dist/index.js",
@@ -37,7 +37,7 @@
"ai"
],
"dependencies": {
- "@anthropic-ai/sdk": "^0.68.0",
+ "@anthropic-ai/sdk": "^0.69.0",
"@google/genai": "^1.28.0",
"@hyperbrowser/sdk": "^0.46.0",
"@inquirer/prompts": "^7.4.1",
diff --git a/scripts/gasPrice.ts b/scripts/gasPrice.ts
index a6310d27..6d2615b9 100644
--- a/scripts/gasPrice.ts
+++ b/scripts/gasPrice.ts
@@ -25,6 +25,12 @@ async function runWorkflow() {
model: "claude-sonnet-4-0",
},
debug: true,
+ cdpActions: true,
+ debugOptions: {
+ cdpSessions: true,
+ traceWait: true,
+ profileDomCapture: true,
+ },
});
// Get the page instance
@@ -64,18 +70,18 @@ async function runWorkflow() {
await page.aiAction(`click the Gas button`);
// Scroll: Scrolled down 300 pixels
- await page.aiAction(`Scroll down 300 pixels`);
+ await page.aiAction(`Scroll down to bottom`);
// Scroll: Scrolled down 500 pixels
- await page.aiAction(`Scroll down 500 pixels`);
+ await page.aiAction(`Scroll down to bottom`);
// Scroll: Scrolled down 800 pixels
- await page.aiAction(`Scroll down 800 pixels`);
+ await page.aiAction(`Scroll down to bottom`);
// Step 11: Extract data
- console.log(
- `Extracting: Extract all gas stations shown in the results list with their names, addresses, and regular gas prices per gallon`
- );
+ // console.log(
+ // `Extracting: Extract all gas stations shown in the results list with their names, addresses, and regular gas prices per gallon`
+ // );
// const extractedData11 = await page.extract({
// instruction: `Extract all gas stations shown in the results list with their names, addresses, and regular gas prices per gallon`,
// schema: z.object({
diff --git a/scripts/test-page-ai.ts b/scripts/test-page-ai.ts
index ee1030d9..48a9ba19 100644
--- a/scripts/test-page-ai.ts
+++ b/scripts/test-page-ai.ts
@@ -3,46 +3,71 @@ import dotenv from "dotenv";
dotenv.config();
-// const agent = new HyperAgent();
-
-// (async () => {
-// const page = await agent.newPage();
-// page.ai(
-// "Go to https://flights.google.com and find a round-trip flight from Rio de Janeiro to Los Angeles, leaving on November 11, 2025, and returning on November 22, 2025, and select the option with the least carbon dioxide emissions."
-// );
-
-// const page2 = await agent.newPage();
-// await page2.goto("https://maps.google.com");
-// page2.ai("Find the nearest restaurant to the current page");
-// })();
-
(async () => {
const agent = new HyperAgent({
llm: {
provider: "anthropic",
model: "claude-sonnet-4-0",
},
- // browserProvider: "Hyperbrowser",
+ // llm: {
+ // provider: "openai",
+ // model: "gpt-4o",
+ // },
+ browserProvider: "Hyperbrowser",
debug: true,
+ debugOptions: {
+ cdpSessions: true,
+ traceWait: true,
+ profileDomCapture: true,
+ // structuredSchema: true,
+ },
+ cdpActions: true,
});
+
const page = await agent.newPage();
- page.goto("https://flights.google.com");
- await page.aiAction("click source location box");
- await page.aiAction("type 'Rio de Janeiro' into the source location box");
- await page.aiAction("press enter");
- await page.aiAction("click destination location box");
- await page.aiAction("type 'Los Angeles' into the destination location box");
- await page.aiAction("press enter");
- await page.aiAction("click the departure date box");
- await page.aiAction(
- "fill 12/01/2025 into the departure date box"
+ page.ai(
+ "Go to https://flights.google.com and find a round-trip flight from Rio de Janeiro to Los Angeles, leaving on November Dec 11, 2025, and returning on Dec 22, 2025, and select the option with the least carbon dioxide emissions.",
+ {
+ useDomCache: true,
+ enableDomStreaming: true,
+ }
);
- await page.aiAction("click the return date box");
- await page.aiAction("fill 12/22/2025 into the return date box");
- await page.aiAction("click the search button");
- await page.aiAction("click the first flight option");
- // const page2 = await agent.newPage();
- // await page2.goto("https://maps.google.com");
- // page2.ai("Find the nearest restaurant to the current page");
+ const page2 = await agent.newPage();
+ await page2.goto("https://maps.google.com");
+ page2.ai("Find the nearest restaurant to the current page", {
+ useDomCache: true,
+ enableDomStreaming: true,
+ });
})();
+
+// (async () => {
+// const agent = new HyperAgent({
+// llm: {
+// provider: "anthropic",
+// model: "claude-sonnet-4-0",
+// },
+// // browserProvider: "Hyperbrowser",
+// debug: true,
+// });
+// const page = await agent.newPage();
+// page.goto("https://flights.google.com");
+// await page.aiAction("click source location box");
+// await page.aiAction("type 'Rio de Janeiro' into the source location box");
+// await page.aiAction("press enter");
+// await page.aiAction("click destination location box");
+// await page.aiAction("type 'Los Angeles' into the destination location box");
+// await page.aiAction("press enter");
+// await page.aiAction("click the departure date box");
+// await page.aiAction(
+// "fill 12/01/2025 into the departure date box"
+// );
+// await page.aiAction("click the return date box");
+// await page.aiAction("fill 12/22/2025 into the return date box");
+// await page.aiAction("click the search button");
+// await page.aiAction("click the first flight option");
+
+// // const page2 = await agent.newPage();
+// // await page2.goto("https://maps.google.com");
+// // page2.ai("Find the nearest restaurant to the current page");
+// })();
diff --git a/src/agent/actions/act-element.ts b/src/agent/actions/act-element.ts
index 9390147d..a961565e 100644
--- a/src/agent/actions/act-element.ts
+++ b/src/agent/actions/act-element.ts
@@ -1,32 +1,42 @@
import { z } from "zod";
import { ActionContext, ActionOutput, AgentActionDefinition } from "@/types";
-import { examineDom } from "../examine-dom";
-import { executePlaywrightMethod } from "../shared/execute-playwright-method";
-import { getElementLocator } from "../shared/element-locator";
import { AGENT_ELEMENT_ACTIONS } from "../shared/action-restrictions";
+import { performAction } from "./shared/perform-action";
+
+const methodSchema = z
+ .enum(AGENT_ELEMENT_ACTIONS)
+ .describe(
+ "Method to execute (click, fill, type, press, selectOptionFromDropdown, check, uncheck, hover, scrollToElement, scrollToPercentage, nextChunk, prevChunk)."
+ );
const ActElementAction = z
.object({
instruction: z
.string()
+ .describe("Short explanation of why this action is needed."),
+ elementId: z
+ .string()
+ .min(1)
+ .describe(
+ 'Encoded element identifier from the DOM listing (format "frameIndex-backendNodeId", e.g., "0-5125").'
+ ),
+ method: methodSchema.describe(
+ "CDP/Playwright method to invoke (click, fill, type, press, selectOptionFromDropdown, check, uncheck, hover, scrollToElement, scrollToPercentage, nextChunk, prevChunk)."
+ ),
+ arguments: z
+ .array(z.string())
.describe(
- "Describe the action in a short, specific phrase that mentions the element type.\n\n" +
- "Supported actions: click, fill, type, press, selectOptionFromDropdown, check, uncheck, hover, scrollTo, nextChunk, prevChunk\n\n" +
- "Examples:\n" +
- "- click the Login button\n" +
- "- fill 'user@example.com' into email field\n" +
- "- type 'search query' into search box\n" +
- "- press Enter\n" +
- "- select 'California' from state dropdown\n" +
- "- check the terms checkbox\n" +
- "- uncheck the newsletter checkbox\n" +
- "- hover over profile menu\n" +
- "- scroll to 50%\n" +
- "- scroll down one page\n" +
- "- scroll up one page"
+ "Arguments for the method (e.g., text to fill, key to press, scroll target). Use an empty array when no arguments are required."
+ ),
+ confidence: z
+ .number()
+ .describe(
+ "LLM-estimated confidence (0-1). Used for debugging/telemetry; execution does not depend on it."
),
})
- .describe("Perform a single action on an element using natural language");
+ .describe(
+ "Perform a single action on an element by referencing an encoded ID from the DOM listing."
+ );
type ActElementActionType = z.infer;
@@ -37,99 +47,7 @@ export const ActElementActionDefinition: AgentActionDefinition = {
ctx: ActionContext,
action: ActElementActionType
): Promise {
- const { instruction } = action;
-
- // DOM state is provided by agent loop in ctx.domState
- // NO DOM FETCHING HERE - agent loop handles that
-
- // Convert elements map for examineDom
- const elementMap = new Map(
- Array.from(ctx.domState.elements).map(([k, v]) => [String(k), v])
- );
-
- // Call examineDom with current DOM state
- const examineResult = await examineDom(
- instruction,
- {
- tree: ctx.domState.domState,
- xpathMap: ctx.domState.xpathMap || {},
- elements: elementMap,
- url: ctx.page.url(),
- },
- ctx.llm
- );
-
- // Check if element was found
- if (!examineResult || examineResult.elements.length === 0) {
- return {
- success: false,
- message: `Failed to execute "${instruction}": Element not found on page`,
- };
- }
-
- const element = examineResult.elements[0];
- const method = element.method;
- const args = element.arguments || [];
-
- // Store debug info about selected element
- const debugInfo = ctx.debug
- ? {
- selectedElement: {
- elementId: element.elementId,
- confidence: element.confidence,
- description: element.description,
- method: method,
- arguments: args,
- },
- allCandidates: examineResult.elements.map((e) => ({
- elementId: e.elementId,
- confidence: e.confidence,
- description: e.description,
- })),
- }
- : undefined;
-
- // Validate action is allowed
- if (!AGENT_ELEMENT_ACTIONS.includes(method)) {
- return {
- success: false,
- message: `Action "${method}" not allowed. Allowed actions: ${AGENT_ELEMENT_ACTIONS.join(
- ", "
- )}`,
- debug: debugInfo,
- };
- }
-
- try {
- // Get Playwright locator using shared utility
- const { locator } = await getElementLocator(
- element.elementId,
- ctx.domState.xpathMap,
- ctx.page,
- ctx.domState.frameMap,
- !!ctx.debugDir
- );
-
- // Execute Playwright method using shared utility
- await executePlaywrightMethod(method, args, locator, {
- clickTimeout: ctx.actionConfig?.clickElement?.timeout ?? 3500,
- debug: !!ctx.debugDir,
- });
-
- return {
- success: true,
- message: `Successfully executed: ${instruction}`,
- debug: debugInfo,
- };
- } catch (error) {
- const errorMessage =
- error instanceof Error ? error.message : String(error);
- return {
- success: false,
- message: `Failed to execute "${instruction}": ${errorMessage}`,
- debug: debugInfo,
- };
- }
+ return performAction(ctx, action);
},
pprintAction: function (params: ActElementActionType): string {
return `Act: ${params.instruction}`;
diff --git a/src/agent/actions/complete-validator.ts b/src/agent/actions/complete-validator.ts
deleted file mode 100644
index cd14cda4..00000000
--- a/src/agent/actions/complete-validator.ts
+++ /dev/null
@@ -1,52 +0,0 @@
-import { z } from "zod";
-import { ActionContext, ActionOutput, AgentActionDefinition } from "@/types";
-
-export const CompletionValidateAction = z
- .object({
- task: z
- .string()
- .describe("The detailed description of the task to complete."),
- completionCriteria: z.array(
- z.object({
- subTask: z
- .string()
- .describe("The description of the specific sub task of the task."),
- subTaskSatisfied: z
- .boolean()
- .describe("Is the specific sub task of the task completed."),
- subTaskSatisfiedReason: z
- .string()
- .describe(
- "How and why has this subtask been marked as completed (if completed). Provide the result as well if this response required an action, and that action produced a result."
- ),
- })
- ),
- })
- .describe(
- `Must run this before issuing the final complete action to validate that the task is completed.
- Evaluate if all the sub parts of the task are completed, and so if the task itself is completed. If you don't run this step, you will be heavily penalized.`
- );
-
-export type CompleteValidateActionType = z.infer<
- typeof CompletionValidateAction
->;
-
-export const CompletionValidateActionDefinition: AgentActionDefinition = {
- type: "taskCompleteValidation",
- actionParams: CompletionValidateAction,
- run: async (
- ctx: ActionContext,
- action: CompleteValidateActionType
- ): Promise => {
- const completionCriteria = action.completionCriteria
- .map(
- (subTask) =>
- `subTask:${subTask.subTask} || condition satisfied: ${subTask.subTaskSatisfied}`
- )
- .join("\n");
- return {
- success: true,
- message: `Task Completion Report: \ntask:${action.task} \nsubtasks: \n${completionCriteria}`,
- };
- },
-};
diff --git a/src/agent/actions/extract.ts b/src/agent/actions/extract.ts
index fcc6b9cb..8fb366f9 100644
--- a/src/agent/actions/extract.ts
+++ b/src/agent/actions/extract.ts
@@ -2,6 +2,7 @@ import { z } from "zod";
import { ActionContext, ActionOutput, AgentActionDefinition } from "@/types";
import { parseMarkdown } from "@/utils/html-to-markdown";
import fs from "fs";
+import { getCDPClient } from "@/cdp";
export const ExtractAction = z
.object({
@@ -26,20 +27,18 @@ export const ExtractActionDefinition: AgentActionDefinition = {
const objective = action.objective;
// Take a screenshot of the page
- const cdpSession = await ctx.page.context().newCDPSession(ctx.page);
- let screenshot;
- try {
- screenshot = await cdpSession.send("Page.captureScreenshot");
+ const cdpClient = await getCDPClient(ctx.page);
+ const cdpSession = await cdpClient.acquireSession("screenshot");
+ const screenshot = await cdpSession.send<{ data: string }>(
+ "Page.captureScreenshot"
+ );
- // Save screenshot to debug dir if exists
- if (ctx.debugDir) {
- fs.writeFileSync(
- `${ctx.debugDir}/extract-screenshot.png`,
- Buffer.from(screenshot.data, "base64")
- );
- }
- } finally {
- await cdpSession.detach();
+ // Save screenshot to debug dir if exists
+ if (ctx.debugDir) {
+ fs.writeFileSync(
+ `${ctx.debugDir}/extract-screenshot.png`,
+ Buffer.from(screenshot.data, "base64")
+ );
}
// Trim markdown to stay within token limit
@@ -63,7 +62,7 @@ export const ExtractActionDefinition: AgentActionDefinition = {
content: [
{
type: "text",
- text: `Extract the following information from the page according to this objective: "${objective}"\n\nPage content:\n${trimmedMarkdown}\nHere is as screenshot of the page:\n`,
+ text: `Extract the following information from the page according to this objective: "${objective}"\n\nPage content:\n${trimmedMarkdown}\nHere is a screenshot of the page:\n`,
},
{
type: "image",
diff --git a/src/agent/actions/index.ts b/src/agent/actions/index.ts
index 08520f88..f5805f06 100644
--- a/src/agent/actions/index.ts
+++ b/src/agent/actions/index.ts
@@ -29,8 +29,8 @@ export class ActionNotFoundError extends Error {
const DEFAULT_ACTIONS = [
// Navigation actions
GoToURLActionDefinition,
- PageBackActionDefinition,
- PageForwardActionDefinition,
+ // PageBackActionDefinition,
+ // PageForwardActionDefinition,
RefreshPageActionDefinition,
// Element interaction (natural language)
@@ -38,7 +38,7 @@ const DEFAULT_ACTIONS = [
// Other actions
ExtractActionDefinition,
- ThinkingActionDefinition,
+ // ThinkingActionDefinition, // Disabled: agents waste steps thinking instead of acting; thoughts field already provides reasoning
WaitActionDefinition,
];
diff --git a/src/agent/actions/shared/perform-action.ts b/src/agent/actions/shared/perform-action.ts
new file mode 100644
index 00000000..91892308
--- /dev/null
+++ b/src/agent/actions/shared/perform-action.ts
@@ -0,0 +1,155 @@
+import { performance } from "perf_hooks";
+import { ActionContext, ActionOutput } from "@/types";
+import type { ResolvedCDPElement, CDPActionMethod } from "@/cdp";
+import { isEncodedId, type EncodedId } from "@/context-providers/a11y-dom/types";
+import { getElementLocator } from "../../shared/element-locator";
+import { executePlaywrightMethod } from "../../shared/execute-playwright-method";
+
+export interface PerformActionParams {
+ elementId: string;
+ method: string;
+ arguments?: string[];
+ instruction: string;
+ confidence?: number;
+}
+
+/**
+ * Performs a single action on an element
+ * Consolidates logic for choosing between CDP and Playwright execution paths
+ */
+export async function performAction(
+ ctx: ActionContext,
+ params: PerformActionParams
+): Promise {
+ const {
+ instruction,
+ elementId,
+ method,
+ arguments: methodArgs = [],
+ confidence,
+ } = params;
+
+ if (!isEncodedId(elementId)) {
+ return {
+ success: false,
+ message: `Failed to execute "${instruction}": elementId "${elementId}" is not in encoded format (frameIndex-backendNodeId).`,
+ };
+ }
+
+ const encodedId = elementId;
+ const elementMetadata = ctx.domState.elements.get(encodedId);
+ if (!elementMetadata) {
+ return {
+ success: false,
+ message: `Failed to execute "${instruction}": elementId "${elementId}" not present in current DOM.`,
+ };
+ }
+
+ const timings: Record | undefined = ctx.debug ? {} : undefined;
+ const debugInfo =
+ ctx.debug && elementMetadata
+ ? {
+ requestedAction: {
+ elementId,
+ method,
+ arguments: methodArgs,
+ confidence,
+ instruction,
+ },
+ elementMetadata,
+ ...(timings ? { timings } : {}),
+ }
+ : undefined;
+
+ const shouldUseCDP =
+ !!ctx.cdp && ctx.cdpActions !== false && !!ctx.domState.backendNodeMap;
+
+ if (shouldUseCDP) {
+ const resolvedElementsCache = new Map();
+ try {
+ const resolveStart = performance.now();
+ const resolved = await ctx.cdp!.resolveElement(encodedId, {
+ page: ctx.page,
+ cdpClient: ctx.cdp!.client,
+ backendNodeMap: ctx.domState.backendNodeMap,
+ xpathMap: ctx.domState.xpathMap,
+ frameMap: ctx.domState.frameMap,
+ resolvedElementsCache,
+ frameContextManager: ctx.cdp!.frameContextManager,
+ debug: ctx.debug,
+ strictFrameValidation: true,
+ });
+ if (timings) {
+ timings.resolveElementMs = Math.round(performance.now() - resolveStart);
+ }
+
+ const dispatchStart = performance.now();
+ await ctx.cdp!.dispatchCDPAction(method as CDPActionMethod, methodArgs, {
+ element: {
+ ...resolved,
+ xpath: ctx.domState.xpathMap?.[encodedId],
+ },
+ boundingBox: ctx.domState.boundingBoxMap?.get(encodedId) ?? undefined,
+ preferScriptBoundingBox: ctx.cdp!.preferScriptBoundingBox,
+ debug: ctx.cdp?.debug ?? ctx.debug,
+ });
+ if (timings) {
+ timings.dispatchMs = Math.round(performance.now() - dispatchStart);
+ }
+
+ return {
+ success: true,
+ message: `Successfully executed: ${instruction}`,
+ debug: debugInfo,
+ };
+ } catch (error) {
+ const errorMessage =
+ error instanceof Error ? error.message : String(error);
+ return {
+ success: false,
+ message: `Failed to execute "${instruction}": ${errorMessage}`,
+ debug: debugInfo,
+ };
+ }
+ }
+
+ try {
+ // Get Playwright locator using shared utility
+ const locatorStart = performance.now();
+ const { locator } = await getElementLocator(
+ elementId,
+ ctx.domState.xpathMap,
+ ctx.page,
+ ctx.domState.frameMap,
+ !!ctx.debugDir
+ );
+ if (timings) {
+ timings.locatorMs = Math.round(performance.now() - locatorStart);
+ }
+
+ // Execute Playwright method using shared utility
+ const pwStart = performance.now();
+ await executePlaywrightMethod(method, methodArgs, locator, {
+ clickTimeout: 3500,
+ debug: !!ctx.debugDir,
+ });
+ if (timings) {
+ timings.playwrightActionMs = Math.round(performance.now() - pwStart);
+ }
+
+ return {
+ success: true,
+ message: `Successfully executed: ${instruction}`,
+ debug: debugInfo,
+ };
+ } catch (error) {
+ const errorMessage =
+ error instanceof Error ? error.message : String(error);
+ return {
+ success: false,
+ message: `Failed to execute "${instruction}": ${errorMessage}`,
+ debug: debugInfo,
+ };
+ }
+}
+
diff --git a/src/agent/actions/thinking.ts b/src/agent/actions/thinking.ts
index 27d51996..68fc7503 100644
--- a/src/agent/actions/thinking.ts
+++ b/src/agent/actions/thinking.ts
@@ -3,12 +3,13 @@ import { ActionContext, AgentActionDefinition } from "@/types";
export const ThinkingAction = z
.object({
- thought: z
+ plan: z
.string()
.describe(
- "Think about what your current course of action, and your future steps, and what difficulties you might encounter, and how you'd tackle them."
+ "Describe your strategic plan for the next steps, including potential obstacles and how you'll tackle them."
),
})
+ .required()
.describe(
`Think about a course of action. Think what your current task is, what your next should be, and how you would possibly do that. This step is especially useful if performing a complex task, and/or working on a visually complex page (think nodes > 300).`
);
@@ -19,13 +20,13 @@ export const ThinkingActionDefinition: AgentActionDefinition = {
type: "thinking" as const,
actionParams: ThinkingAction,
run: async (ctx: ActionContext, action: ThinkingActionType) => {
- const { thought } = action;
+ const { plan } = action;
return {
success: true,
- message: `A simple thought process about your next steps. You thought about: ${thought}`,
+ message: `A simple thought process about your next steps. You planned: ${plan}`,
};
},
- pprintAction: function(params: ThinkingActionType): string {
- return `Think about: "${params.thought}"`;
+ pprintAction: function (params: ThinkingActionType): string {
+ return `Think about: "${params.plan}"`;
},
};
diff --git a/src/agent/actions/wait.ts b/src/agent/actions/wait.ts
index cfc04c43..db87e2f2 100644
--- a/src/agent/actions/wait.ts
+++ b/src/agent/actions/wait.ts
@@ -4,9 +4,11 @@ import { waitForSettledDOM } from "@/utils/waitForSettledDOM";
const WaitAction = z
.object({
- reason: z.string().describe(
- "Explain why you cannot confidently take an action right now (e.g., 'Page is still loading', 'Expected element not visible yet', 'Waiting for dynamic content to appear', 'Page may still be transitioning')"
- ),
+ reason: z
+ .string()
+ .describe(
+ "Explain why you cannot confidently take an action right now (e.g., 'Page is still loading', 'Expected element not visible yet', 'Waiting for dynamic content to appear', 'Page may still be transitioning')"
+ ),
})
.describe("Use this action when you are not confident enough to take a meaningful action. The page may still be loading, elements may not be visible yet, or the page state may be unclear. The system will wait for the DOM to settle and give you a fresh view.");
diff --git a/src/agent/examine-dom/prompts.ts b/src/agent/examine-dom/prompts.ts
index 2e9e267c..a186332c 100644
--- a/src/agent/examine-dom/prompts.ts
+++ b/src/agent/examine-dom/prompts.ts
@@ -26,7 +26,8 @@ export function buildActionInstruction(action: string): string {
"fill",
"type",
"press",
- "scrollTo",
+ "scrollToElement",
+ "scrollToPercentage",
"nextChunk",
"prevChunk",
"selectOptionFromDropdown",
@@ -39,8 +40,8 @@ export function buildActionInstruction(action: string): string {
Provide an action for this element such as ${supportedActions.join(", ")}, or any other playwright locator method. Remember that to users, buttons and links look the same in most cases.
If the action is completely unrelated to a potential action to be taken on the page, return an empty array.
ONLY return one action. If multiple actions are relevant, return the most relevant one.
-For scroll actions (scrollTo, nextChunk, prevChunk), prefer elements marked as "scrollable" in their role. These have been automatically detected as scrollable containers. If no scrollable elements are available, choose the html element as a fallback.
-If the user is asking to scroll to a position on the page, e.g., 'halfway' or 0.75, etc, you must return the scrollTo method with the argument formatted as the correct percentage, e.g., '50%' or '75%', etc.
+For scroll actions (scrollToElement, scrollToPercentage, nextChunk, prevChunk), prefer elements marked as "scrollable" in their role. These have been automatically detected as scrollable containers. If no scrollable elements are available, choose the html element as a fallback.
+Use scrollToElement (no arguments) when the request is to reveal a specific section or component. Use scrollToPercentage (with a percentage argument like "50%" or "75%") only when the user explicitly mentions a relative position on the page.
If the user is asking to scroll to the next chunk/previous chunk, choose the nextChunk/prevChunk method. No arguments are required here.
If the action implies a key press, e.g., 'press enter', 'press a', 'press space', etc., always choose the press method with the appropriate key as argument β e.g. 'a', 'Enter', 'Space'. Do not choose a click action on an on-screen keyboard. Capitalize the first character like 'Enter', 'Tab', 'Escape' only for special keys.
If the action implies choosing an option from a dropdown, AND the corresponding element is a 'select' element, choose the selectOptionFromDropdown method. The argument should be the text of the option to select.
diff --git a/src/agent/examine-dom/schema.ts b/src/agent/examine-dom/schema.ts
index cccba95e..14ae779b 100644
--- a/src/agent/examine-dom/schema.ts
+++ b/src/agent/examine-dom/schema.ts
@@ -1,4 +1,5 @@
-import { z } from 'zod';
+import { z } from "zod";
+import { AGENT_ELEMENT_ACTIONS } from "../shared/action-restrictions";
/**
* Zod schema for a single element match result
@@ -16,20 +17,8 @@ export const ExamineDomResultSchema = z.object({
.max(1)
.describe('Confidence score 0-1 indicating match quality'),
method: z
- .enum([
- 'click',
- 'fill',
- 'type',
- 'press',
- 'scrollTo',
- 'nextChunk',
- 'prevChunk',
- 'selectOptionFromDropdown',
- 'hover',
- 'check',
- 'uncheck',
- ])
- .default('click')
+ .enum(AGENT_ELEMENT_ACTIONS)
+ .default("click")
.describe('Suggested Playwright method to use'),
arguments: z
.array(z.string())
diff --git a/src/agent/examine-dom/types.ts b/src/agent/examine-dom/types.ts
index 904d688f..2799846a 100644
--- a/src/agent/examine-dom/types.ts
+++ b/src/agent/examine-dom/types.ts
@@ -5,6 +5,7 @@
import { ExamineDomResultSchema } from "./schema";
import { z } from "zod";
+import type { AccessibilityNode } from "@/context-providers/a11y-dom/types";
/**
* Playwright methods that can be performed on elements
@@ -14,7 +15,8 @@ export type PlaywrightMethod =
| "fill"
| "type"
| "press"
- | "scrollTo"
+ | "scrollToElement"
+ | "scrollToPercentage"
| "nextChunk"
| "prevChunk"
| "selectOptionFromDropdown"
@@ -35,7 +37,7 @@ export interface ExamineDomContext {
xpathMap: Record;
/** Map of elementIds to accessibility node objects */
- elements: Map;
+ elements: Map;
/** Current page URL */
url: string;
diff --git a/src/agent/index.ts b/src/agent/index.ts
index c7e61e0c..acc9eedd 100644
--- a/src/agent/index.ts
+++ b/src/agent/index.ts
@@ -9,6 +9,7 @@ import {
} from "@/types/config";
import { HyperAgentLLM, createLLMClient } from "@/llm/providers";
import {
+ ActionContext,
ActionType,
AgentActionDefinition,
endTaskStatuses,
@@ -29,16 +30,24 @@ import {
} from "../browser-providers";
import { HyperagentError } from "./error";
import { findElementWithInstruction } from "./shared/find-element";
-import { executePlaywrightMethod } from "./shared/execute-playwright-method";
-import { getElementLocator } from "./shared/element-locator";
-import { A11yDOMState } from "../context-providers/a11y-dom/types";
+import {
+ A11yDOMState,
+ AccessibilityNode,
+ isEncodedId,
+} from "../context-providers/a11y-dom/types";
import { MCPClient } from "./mcp/client";
import { runAgentTask } from "./tools/agent";
import { HyperPage, HyperVariable } from "../types/agent/types";
import { z } from "zod";
import { ErrorEmitter } from "../utils";
import { waitForSettledDOM } from "@/utils/waitForSettledDOM";
+import { performance } from "perf_hooks";
import { ExamineDomResult } from "./examine-dom/types";
+import { disposeAllCDPClients, resolveElement, dispatchCDPAction } from "@/cdp";
+import { markDomSnapshotDirty } from "@/context-providers/a11y-dom/dom-cache";
+import { setDebugOptions } from "@/debug/options";
+import { initializeRuntimeContext } from "./shared/runtime-context";
+import { performAction } from "./actions/shared/perform-action";
export class HyperAgent {
// aiAction configuration constants
@@ -61,7 +70,7 @@ export class HyperAgent {
: LocalBrowserProvider;
private browserProviderType: T;
private actions: Array = [...DEFAULT_ACTIONS];
- private actionConfig: HyperAgentConfig["actionConfig"];
+ private cdpActionsEnabled: boolean;
public browser: Browser | null = null;
public context: BrowserContext | null = null;
@@ -100,6 +109,9 @@ export class HyperAgent {
}
this.browserProviderType = (params.browserProvider ?? "Local") as T;
+ setDebugOptions(params.debugOptions, this.debug);
+
+ // TODO(Phase4): This legacy provider branch will be replaced by connector configs.
this.browserProvider = (
this.browserProviderType === "Hyperbrowser"
? new HyperbrowserProvider({
@@ -114,7 +126,7 @@ export class HyperAgent {
}
this.debug = params.debug ?? false;
- this.actionConfig = params.actionConfig;
+ this.cdpActionsEnabled = params.cdpActions ?? true;
this.errorEmitter = new ErrorEmitter();
}
@@ -287,6 +299,9 @@ export class HyperAgent {
* Close the agent and all associated resources
*/
public async closeAgent(): Promise {
+ await disposeAllCDPClients().catch((error) => {
+ console.warn("[HyperAgent] Failed to dispose CDP clients:", error);
+ });
for (const taskId in this.tasks) {
const task = this.tasks[taskId];
if (!endTaskStatuses.has(task.status)) {
@@ -381,18 +396,19 @@ export class HyperAgent {
steps: [],
};
this.tasks[taskId] = taskState;
+ const mergedParams = params ?? {};
runAgentTask(
{
llm: this.llm,
- actions: this.getActions(params?.outputSchema),
+ actions: this.getActions(mergedParams.outputSchema),
tokenLimit: this.tokenLimit,
debug: this.debug,
mcpClient: this.mcpClient,
variables: this._variables,
- actionConfig: this.actionConfig,
+ cdpActions: this.cdpActionsEnabled,
},
taskState,
- params
+ mergedParams
).catch((error: Error) => {
// Retrieve the correct state to update
const failedTaskState = this.tasks[taskId];
@@ -432,18 +448,19 @@ export class HyperAgent {
};
this.tasks[taskId] = taskState;
try {
+ const mergedParams = params ?? {};
return await runAgentTask(
{
llm: this.llm,
- actions: this.getActions(params?.outputSchema),
+ actions: this.getActions(mergedParams?.outputSchema),
tokenLimit: this.tokenLimit,
debug: this.debug,
mcpClient: this.mcpClient,
variables: this._variables,
- actionConfig: this.actionConfig,
+ cdpActions: this.cdpActionsEnabled,
},
taskState,
- params
+ mergedParams
);
} catch (error) {
taskState.status = TaskStatus.FAILED;
@@ -471,7 +488,7 @@ export class HyperAgent {
): Promise<{
element: ExamineDomResult;
domState: A11yDOMState;
- elementMap: Map;
+ elementMap: Map;
llmResponse: { rawText: string; parsed: unknown };
}> {
// Delegate to shared utility
@@ -529,13 +546,6 @@ export class HyperAgent {
);
}
- /**
- * Write debug data for aiAction execution
- * Captures screenshot, DOM state, and execution details for debugging
- *
- * @param params Debug data parameters
- * @returns Promise that resolves when debug data is written
- */
private async writeDebugData(params: {
instruction: string;
page: Page;
@@ -543,7 +553,7 @@ export class HyperAgent {
domState: Awaited<
ReturnType
> | null;
- elementMap: Map | null;
+ elementMap: Map | null;
element?: {
elementId: string;
method: string;
@@ -629,7 +639,7 @@ export class HyperAgent {
* @returns Array of interactive elements with id, role, and label
*/
private collectInteractiveElements(
- elementMap: Map,
+ elementMap: Map,
limit: number = 20
): Array<{ id: string; role: string; label: string }> {
// Group elements by frame
@@ -639,11 +649,7 @@ export class HyperAgent {
>();
for (const [id, elem] of elementMap) {
- // Type guard: ensure elem is an object with expected properties
- if (!elem || typeof elem !== "object") continue;
-
- const node = elem as Record;
- const role = typeof node.role === "string" ? node.role : undefined;
+ const role = elem.role;
if (
role &&
@@ -658,11 +664,7 @@ export class HyperAgent {
"menuitem",
].includes(role)
) {
- const name = typeof node.name === "string" ? node.name : undefined;
- const description =
- typeof node.description === "string" ? node.description : undefined;
- const value = typeof node.value === "string" ? node.value : undefined;
- const label = name || description || value || "";
+ const label = elem.name || elem.description || elem.value || "";
if (label) {
// Extract frame index from ID (format: "frameIndex-backendNodeId")
@@ -711,19 +713,21 @@ export class HyperAgent {
*/
private async executeSingleAction(
instruction: string,
- page: Page
+ page: Page,
+ _params?: TaskParams
): Promise {
+ const actionStart = performance.now();
const startTime = new Date().toISOString();
-
if (this.debug) {
console.log(`[aiAction] Instruction: ${instruction}`);
}
let domState: A11yDOMState | null = null;
- let elementMap: Map | null = null;
+ let elementMap: Map | null = null;
try {
// Find element with retry logic
+ const findStart = performance.now();
const {
element,
domState: foundDomState,
@@ -739,6 +743,11 @@ export class HyperAgent {
domState = foundDomState;
elementMap = foundElementMap;
+ logPerf(
+ this.debug,
+ "[Perf][executeSingleAction] findElementWithRetry",
+ findStart
+ );
if (this.debug) {
console.log(`[aiAction] Found element: ${element.elementId}`);
@@ -746,16 +755,6 @@ export class HyperAgent {
console.log(`[aiAction] Arguments:`, element.arguments);
}
- // Get Playwright locator for the element (xpath is already trimmed by getElementLocator)
- const { locator, xpath } = await getElementLocator(
- element.elementId,
- domState.xpathMap,
- page,
- domState.frameMap,
- this.debug
- );
-
- // Execute the Playwright method
if (!element.method) {
throw new HyperagentError(
"Element method is missing from LLM response",
@@ -764,13 +763,83 @@ export class HyperAgent {
}
const method = element.method;
const args = element.arguments || [];
- await executePlaywrightMethod(method, args, locator, {
- clickTimeout: HyperAgent.AIACTION_CONFIG.CLICK_TIMEOUT,
+ if (!isEncodedId(element.elementId)) {
+ throw new HyperagentError(
+ `Element ID "${element.elementId}" is not in encoded format (frameIndex-backendNodeId).`,
+ 400
+ );
+ }
+ let actionXPath: string | undefined;
+
+ // Use shared runtime context
+ const { cdpClient, frameContextManager } = await initializeRuntimeContext(
+ page,
+ this.debug
+ );
+
+ // Create a context object compatible with performAction
+ // We need to mock the ActionContext shape since performAction expects it
+ // but we don't have a full AgentCtx/TaskState here
+ const actionContext: ActionContext = {
+ domState,
+ page,
+ tokenLimit: this.tokenLimit,
+ llm: this.llm,
debug: this.debug,
+ // Only provide CDP if enabled
+ cdpActions: this.cdpActionsEnabled,
+ cdp: this.cdpActionsEnabled
+ ? {
+ client: cdpClient,
+ frameContextManager,
+ resolveElement: resolveElement,
+ dispatchCDPAction: dispatchCDPAction,
+ preferScriptBoundingBox: this.debug,
+ debug: this.debug,
+ }
+ : undefined,
+ // These are required by ActionContext but not used by performAction
+ debugDir: undefined,
+ mcpClient: this.mcpClient,
+ variables: Object.values(this._variables),
+ invalidateDomCache: () => markDomSnapshotDirty(page),
+ };
+
+ // Use shared performAction to execute
+ const actionOutput = await performAction(actionContext, {
+ elementId: element.elementId,
+ method,
+ arguments: args,
+ instruction,
+ confidence: 1, // Implicit confidence for single action
});
+ if (
+ actionOutput.debug &&
+ typeof actionOutput.debug === "object" &&
+ "requestedAction" in actionOutput.debug
+ ) {
+ actionXPath = (actionOutput.debug as any).elementMetadata?.xpath;
+ }
+
+ if (!actionOutput.success) {
+ throw new Error(actionOutput.message);
+ }
+
// Wait for DOM to settle after action
+ const waitStart = performance.now();
await waitForSettledDOM(page);
+ markDomSnapshotDirty(page);
+ logPerf(
+ this.debug,
+ "[Perf][executeSingleAction] action execution",
+ actionStart
+ );
+ logPerf(
+ this.debug,
+ "[Perf][executeSingleAction] waitForSettledDOM",
+ waitStart
+ );
// Write debug data on success
await this.writeDebugData({
@@ -783,12 +852,13 @@ export class HyperAgent {
elementId: element.elementId,
method,
arguments: args,
- xpath,
+ xpath: actionXPath,
},
llmResponse,
success: true,
});
+ logPerf(this.debug, "[Perf][executeSingleAction] total", actionStart);
return {
status: TaskStatus.COMPLETED,
steps: [],
@@ -994,8 +1064,8 @@ export class HyperAgent {
const hyperPage = page as HyperPage;
hyperPage.ai = (task: string, params?: TaskParams) =>
this.executeTask(task, params, page);
- hyperPage.aiAction = (instruction: string) =>
- this.executeSingleAction(instruction, page);
+ hyperPage.aiAction = (instruction: string, params?: TaskParams) =>
+ this.executeSingleAction(instruction, page, params);
hyperPage.aiAsync = (task: string, params?: TaskParams) =>
this.executeTaskAsync(task, params, page);
hyperPage.extract = async (task, outputSchema, params) => {
@@ -1017,28 +1087,45 @@ export class HyperAgent {
page
);
if (outputSchema) {
- if (!res.output || res.output === "") {
+ const outputText = res.output;
+ if (typeof outputText !== "string" || outputText === "") {
throw new Error(
`Extract failed: Agent did not complete with output. Task status: ${res.status}. Check debug output for details.`
);
}
- return JSON.parse(res.output as string);
+ return JSON.parse(outputText);
}
- return res.output as string;
+ const outputText = res.output;
+ if (typeof outputText !== "string" || outputText === "") {
+ throw new Error(
+ `Extract failed: Agent did not complete with output. Task status: ${res.status}. Check debug output for details.`
+ );
+ }
+ return outputText;
} else {
const res = await this.executeTask(
"You have to perform a data extraction on the current page. Make sure your final response only contains the extracted content",
taskParams,
page
);
- if (!res.output || res.output === "") {
+ if (typeof res.output !== "string" || res.output === "") {
throw new Error(
`Extract failed: Agent did not complete with output. Task status: ${res.status}. Check debug output for details.`
);
}
- return JSON.parse(res.output as string);
+ return JSON.parse(res.output);
}
};
return hyperPage;
}
}
+
+function logPerf(
+ debug: boolean | undefined,
+ label: string,
+ start: number
+): void {
+ if (!debug) return;
+ const duration = performance.now() - start;
+ console.log(`${label} took ${Math.round(duration)}ms`);
+}
diff --git a/src/agent/messages/builder.ts b/src/agent/messages/builder.ts
index 4072e1d1..852a8580 100644
--- a/src/agent/messages/builder.ts
+++ b/src/agent/messages/builder.ts
@@ -42,16 +42,19 @@ export const buildAgentStepMessages = async (
content: "=== Previous Actions ===\n",
});
for (const step of steps) {
+ const { thoughts, memory, action } = step.agentOutput;
messages.push({
role: "assistant",
- content: JSON.stringify(step.agentOutput),
+ content: `Thoughts: ${thoughts}\nMemory: ${memory}\nAction: ${JSON.stringify(
+ action
+ )}`,
});
- const actionOutput = step.actionOutput;
+ const actionResult = step.actionOutput;
messages.push({
role: "user",
- content: actionOutput.extract
- ? `${actionOutput.message} :\n ${JSON.stringify(actionOutput.extract)}`
- : actionOutput.message,
+ content: actionResult.extract
+ ? `${actionResult.message} :\n ${JSON.stringify(actionResult.extract)}`
+ : actionResult.message,
});
}
}
diff --git a/src/agent/messages/examples-actions.ts b/src/agent/messages/examples-actions.ts
index 7600dc23..3efd91f8 100644
--- a/src/agent/messages/examples-actions.ts
+++ b/src/agent/messages/examples-actions.ts
@@ -1,21 +1,21 @@
export const EXAMPLE_ACTIONS = `# Action Examples
## Element Interaction (actElement)
-- Click: {"type": "actElement", "params": {"instruction": "click the Login button"}}
-- Fill input: {"type": "actElement", "params": {"instruction": "fill 'john@example.com' into email field"}}
-- Type text: {"type": "actElement", "params": {"instruction": "type 'search query' into search box"}}
-- Press key: {"type": "actElement", "params": {"instruction": "press Enter"}}
-- Select dropdown: {"type": "actElement", "params": {"instruction": "select 'California' from state dropdown"}}
-- Check checkbox: {"type": "actElement", "params": {"instruction": "check the terms checkbox"}}
-- Uncheck checkbox: {"type": "actElement", "params": {"instruction": "uncheck the newsletter checkbox"}}
-- Hover: {"type": "actElement", "params": {"instruction": "hover over profile menu"}}
-- Scroll to position: {"type": "actElement", "params": {"instruction": "scroll to 50% of the page"}}
-- Scroll down: {"type": "actElement", "params": {"instruction": "scroll down one page"}}
-- Scroll up: {"type": "actElement", "params": {"instruction": "scroll up one page"}}
+- Click: {"type": "actElement", "params": {"instruction": "Click the Login button to open the form", "elementId": "0-42", "method": "click", "arguments": [], "confidence": 0.92}}
+- Fill input: {"type": "actElement", "params": {"instruction": "Fill in the email field", "elementId": "0-515", "method": "fill", "arguments": ["john@example.com"], "confidence": 0.88}}
+- Type text: {"type": "actElement", "params": {"instruction": "Type in the search box", "elementId": "1-77", "method": "type", "arguments": ["coffee shop"], "confidence": 0.84}}
+- Press key: {"type": "actElement", "params": {"instruction": "Submit the form", "elementId": "0-515", "method": "press", "arguments": ["Enter"], "confidence": 0.73}}
+- Select dropdown: {"type": "actElement", "params": {"instruction": "Choose the California option", "elementId": "2-103", "method": "selectOptionFromDropdown", "arguments": ["California"], "confidence": 0.81}}
+- Check checkbox: {"type": "actElement", "params": {"instruction": "Accept the terms", "elementId": "0-901", "method": "check", "arguments": [], "confidence": 0.79}}
+- Uncheck checkbox: {"type": "actElement", "params": {"instruction": "Disable the newsletter opt-in", "elementId": "0-902", "method": "uncheck", "arguments": [], "confidence": 0.76}}
+- Hover: {"type": "actElement", "params": {"instruction": "Reveal the profile menu", "elementId": "0-1201", "method": "hover", "arguments": [], "confidence": 0.8}}
+- Scroll to element: {"type": "actElement", "params": {"instruction": "Scroll to the pricing section", "elementId": "0-2000", "method": "scrollToElement", "arguments": [], "confidence": 0.7}}
+- Scroll to percentage: {"type": "actElement", "params": {"instruction": "Scroll halfway down the page", "elementId": "0-0", "method": "scrollToPercentage", "arguments": ["50%"], "confidence": 0.7}}
+- Scroll down: {"type": "actElement", "params": {"instruction": "Scroll down one viewport to see more results", "elementId": "0-0", "method": "nextChunk", "arguments": [], "confidence": 0.68}}
+- Scroll up: {"type": "actElement", "params": {"instruction": "Scroll back up", "elementId": "0-0", "method": "prevChunk", "arguments": [], "confidence": 0.66}}
## Other Actions
- Navigate: {"type": "goToUrl", "params": {"url": "https://example.com"}}
- Extract content: {"type": "extract", "params": {"objective": "extract the product price and title"}}
- Wait: {"type": "wait", "params": {"reason": "Waiting for page to finish loading"}}
-- Think: {"type": "thinking", "params": {"thought": "I need to find the login form first before attempting to log in"}}
-- Complete: {"type": "complete", "params": {"success": true, "output": "Task completed successfully"}}`;
+- Complete: {"type": "complete", "params": {"success": true, "text": "Task completed successfully"}}`;
diff --git a/src/agent/messages/input-format.ts b/src/agent/messages/input-format.ts
index 96b0eaf3..d168d424 100644
--- a/src/agent/messages/input-format.ts
+++ b/src/agent/messages/input-format.ts
@@ -22,6 +22,7 @@ export const INPUT_FORMAT = `=== Final Goal ===
* Custom attributes
* Any other valid HTML attributes
* The attributes provide important context about the element's behavior, accessibility, and styling
+- When choosing an element for \`actElement\`, reference the exact \`encodedId\` shown here (for example, \`"0-5125"\`).
=== Previous Actions ===
[The previous steps of the task]
=== Page Screenshot === (only in visual modes)
diff --git a/src/agent/messages/output-format.ts b/src/agent/messages/output-format.ts
index e487a5d4..2dc6f891 100644
--- a/src/agent/messages/output-format.ts
+++ b/src/agent/messages/output-format.ts
@@ -3,9 +3,16 @@ export const OUTPUT_FORMAT = `Your response MUST be in this exact format:
"thoughts": "Your reasoning about the current state and what needs to be done next based on the task goal and previous actions",
"memory": "A summary of successful actions completed so far and the resulting state changes (e.g., 'Clicked login button -> login form appeared', 'Filled email field with user@example.com')",
"action": {
- "type": "The action type to take (actElement, goToUrl, wait, thinking, extract, complete, etc.)",
+ "type": "The action type to take (actElement, goToUrl, wait, extract, complete, etc.)",
"params": {
...Action Arguments...
}
}
-}`
\ No newline at end of file
+}
+
+For actElement:
+- params.instruction -> short explanation of why the action is needed
+- params.elementId -> encoded ID from the DOM listing (e.g., "0-5125")
+- params.method -> one of click, fill, type, press, selectOptionFromDropdown, check, uncheck, hover, scrollToElement, scrollToPercentage, nextChunk, prevChunk
+- params.arguments -> array of arguments for the method (use [] when none are needed)
+- params.confidence -> number between 0 and 1`;
diff --git a/src/agent/messages/simple-system-prompt.ts b/src/agent/messages/simple-system-prompt.ts
deleted file mode 100644
index a4f7f0f0..00000000
--- a/src/agent/messages/simple-system-prompt.ts
+++ /dev/null
@@ -1,114 +0,0 @@
-/**
- * Simplified system prompt for browser automation
- * Key principles:
- * 1. Short and focused
- * 2. Clear task-driven instructions
- * 3. No over-prescription of format
- * 4. Trust LLM to reason naturally
- */
-
-const DATE_STRING = new Date().toLocaleString(undefined, {
- year: "numeric",
- month: "2-digit",
- day: "2-digit",
- weekday: "long",
-});
-
-export const SIMPLE_SYSTEM_PROMPT = `You are a web automation assistant. Your job is to accomplish the user's goal by taking actions on the page.
-
-# Current Context
-- Today's date: ${DATE_STRING}
-- You can see the page structure through an accessibility tree
-- Each action you take will update the page state
-
-# How to Read the Accessibility Tree
-
-The page is shown as a text tree. Each line represents an element:
-
-[elementId] role: accessible name
-
-Example:
-[0-1234] button: Submit
-[0-5678] textbox: Email address
-[0-9012] link: Sign in
-
-- **elementId**: Use this exact ID for actions (e.g., "0-1234")
-- **role**: Element type (button, textbox, link, etc.)
-- **name**: What the element says or does
-
-# Available Actions
-
-You can take ONE action at a time:
-
-1. **clickElement** - Click buttons, links, or clickable elements
- - Use: { "type": "clickElement", "params": { "elementId": "0-1234" } }
-
-2. **inputText** - Type into text fields
- - Use: { "type": "inputText", "params": { "elementId": "0-5678", "text": "hello" } }
-
-3. **selectOption** - Select from dropdowns
- - Use: { "type": "selectOption", "params": { "elementId": "0-9012", "text": "option" } }
-
-4. **scroll** - Scroll the page
- - Use: { "type": "scroll", "params": { "direction": "down" } }
-
-5. **complete** - Mark task as done
- - Use: { "type": "complete", "params": { "output": "result" } }
-
-# Strategy
-
-1. **Understand the goal**: What does the user want?
-2. **Find the element**: Look in the tree for the right element
-3. **Take action**: Use the element's ID with the right action type
-4. **Verify**: After each action, you'll see the new page state
-5. **Complete**: When goal is achieved (or impossible), use complete action
-
-# Important Rules
-
-- Always use the EXACT elementId from the tree (with the dash, like "0-1234")
-- Match elements by their role and name, not just position
-- Take ONE action per turn, then see the result
-- If you can't find what you need, scroll or use complete with explanation
-- If the same action fails 3 times, stop and complete with explanation
-
-# Response Format
-
-Respond with JSON:
-{
- "thoughts": "Your reasoning",
- "memory": "Key information to remember",
- "nextGoal": "What you're trying to do",
- "actions": [
- {
- "type": "actionType",
- "params": { /* parameters */ },
- "actionDescription": "What this does"
- }
- ]
-}
-
-# Example
-
-Task: "Click the login button"
-
-Tree shows:
-[0-100] button: Sign In
-[0-200] button: Sign Up
-
-Your response:
-{
- "thoughts": "User wants to click login button. The 'Sign In' button at [0-100] matches this intent.",
- "memory": "Found Sign In button",
- "nextGoal": "Click the Sign In button",
- "actions": [{
- "type": "clickElement",
- "params": { "elementId": "0-100" },
- "actionDescription": "Clicking Sign In button"
- }]
-}
-
-After clicking, you see new tree, determine if goal achieved, and either:
-- Take next action if needed
-- Use complete action if done
-
-Remember: Stay focused on the user's goal. Take clear, atomic actions. Verify outcomes.`;
diff --git a/src/agent/messages/system-prompt.ts b/src/agent/messages/system-prompt.ts
index 3e8f56ba..e236199b 100644
--- a/src/agent/messages/system-prompt.ts
+++ b/src/agent/messages/system-prompt.ts
@@ -33,15 +33,14 @@ ${OUTPUT_FORMAT}
## Element Interaction
- actElement: Perform action on element using natural language
* Supported interactions: click, fill, type, press, select, check, uncheck, hover
- * Scrolling: scrollTo (scroll element to specific position), scrollNextChunk (scroll down one viewport), scrollPrevChunk (scroll up one viewport)
+ * Scrolling: scrollToElement (scroll the chosen element into view), scrollToPercentage (scroll the page/container to a %), scrollNextChunk (scroll down one viewport), scrollPrevChunk (scroll up one viewport)
* Be specific: mention element type and identifying text
- * Examples: "click the Login button", "fill 'text' into search box", "scroll to 50% of the page", "scroll down one page"
+ * Examples: "click the Login button", "fill 'text' into search box", "scroll to the pricing section", "scroll to 50% of the page", "scroll down one page"
## Utilities
- extract: Extract structured data from the page
- wait: Use when not confident enough to take action (page loading, elements not visible yet)
- complete: Mark task as complete (with success/failure)
-- thinking: Think about your course of action (useful for complex tasks or complex pages)
${EXAMPLE_ACTIONS}
@@ -49,7 +48,7 @@ ${EXAMPLE_ACTIONS}
## Action Rules
- Return EXACTLY ONE action per step
-- Think step-by-step - one operation at a time
+- Execute step-by-step - one operation at a time
- After each action, you will see the result and can decide the next step
- Do not try to predict multiple steps ahead - focus on the immediate next action
- If you're not confident about what action to take (page loading, unclear state), use the "wait" action
@@ -60,6 +59,7 @@ ${EXAMPLE_ACTIONS}
- Be specific in your instructions: mention element type and identifying text
- Examples: "click the Login button", "fill 'user@example.com' into email field"
- The system will automatically find and interact with elements based on your instruction
+- When choosing \`actElement\`, you MUST include the encoded element ID (e.g., "0-5125"), the CDP method (click/fill/etc.), any arguments, and a confidence score. Encoded IDs come directly from the \`=== Elements ===\` section.
## Task Completion
- Only use "complete" when you have fully accomplished everything specified in the task
@@ -68,7 +68,7 @@ ${EXAMPLE_ACTIONS}
## Getting Unstuck
- Avoid getting stuck in loops - do not keep repeating the same actions
-- If stuck, try: going back, starting a new search, opening a new tab, using alternative paths, or using the thinking action
+- If stuck, try: going back, starting a new search, opening a new tab, or using alternative paths
## Special Cases
- Cookies: Accept or close the banner
diff --git a/src/agent/shared/action-restrictions.ts b/src/agent/shared/action-restrictions.ts
index bd69ac27..e536d172 100644
--- a/src/agent/shared/action-restrictions.ts
+++ b/src/agent/shared/action-restrictions.ts
@@ -1,40 +1,10 @@
+import type { CDPActionMethod } from "@/cdp";
+
/**
* Action restrictions for element interactions
* Defines which Playwright methods are allowed for different contexts
*/
-/**
- * Actions allowed for aiAction (executeSingleAction)
- * These are all the Playwright methods that can be executed via natural language
- *
- * aiAction uses a high retry count (10) and is designed for one-off commands
- * where the user directly specifies what they want to do.
- */
-export const AIACTION_ALLOWED_ACTIONS = [
- // Click actions
- "click",
-
- // Input actions
- "fill", // Clear and fill input
- "type", // Type character by character
- "press", // Press keyboard key
-
- // Selection actions
- "selectOptionFromDropdown", // For