Skip to content

Fix/UI tahoe and fixes - #14

Merged
donk8r merged 6 commits into
masterfrom
fix/ui-tahoe-and-fixes
Jun 15, 2026
Merged

Fix/UI tahoe and fixes#14
donk8r merged 6 commits into
masterfrom
fix/ui-tahoe-and-fixes

Conversation

@donk8r

@donk8r donk8r commented Jun 15, 2026

Copy link
Copy Markdown
Member

No description provided.

donk8r and others added 5 commits June 13, 2026 00:30
- Implement Liquid Glass aesthetic with specular rims and box-shadows
- Replace tool details modal with inline expandable step groups
- Add SVG icons and shimmering labels for various tool types
- Enhance animations using cubic-bezier curves and spring motions
- Refine border radii, padding, and typography across all UI components
- Improve dark mode contrast and theme-specific shadow adjustments
- Persist tool execution records and turn durations in history
- Truncate large tool I/O JSON to prevent history file bloat
- Implement explicit SIGTERM shutdown to prevent orphaned sessions
- Improve session recovery with validation and resume flag handling
- Refine address bar visuals and suggestions dropdown behavior
- Update notification toasts, slot cards, and overlay layouts
- Simplify navigation to always open new tabs in background
- Remove new_tab and background flags from NavigateRequest
- Add PointerEvents to click and hover for framework compatibility
- Fix value setter brand-checking for input and textarea elements
- Emulate form submission on Enter key press for input fields
- Scan open shadow roots for elements during snapshots
- Update tool descriptions and README for better clarity

BREAKING CHANGE: browser_navigate no longer accepts new_tab or background parameters
- Implement robust DOM interaction pipeline with actionability checks
- Add support for Shadow DOM, iframes, and occlusion detection
- Implement programmatic handling for JS dialogs and file uploads
- Add observability tools for console messages and network requests
- Create ring buffers for tracking fetch and XHR calls
- Extract JS interaction scripts into a dedicated dom_actions module
- Symlink user tools from ~/.agents/tools into the workspace
- Expand E2E test suite with local fixtures and complex DOM scenarios
- Update MCP tools for improved scrolling and structured navigation
- Refine system prompts with debugging and dialog handling guidance
… suite

Make MCP browser control SOTA-grade and verify it end-to-end.

Navigation
- browser_navigate is always background: opens a new hidden tab (or
  navigates in-place via tab_id) and never moves focus. Dropped the
  new_tab/background flags; browser_switch_tab is the sole focus changer.
- TabManager::open_background keeps is_active truthful; switch/close tab
  state mutates synchronously so an immediate get_tabs is consistent.
  Closing a background tab no longer re-switches or steals focus.

Interaction reliability (dom_actions.rs)
- Unified actionability harness (Playwright-style): resolve-with-retry,
  bounding-box stability, occlusion detection (reports what covers the
  click), full pointer+mouse event sequences, composed/cross-shadow events.
- Fix textarea typing (WebKit brand-check on prototype value setter).
- Enter on a form input submits like a real keypress, with a guarded
  fallback for sites that preventDefault then ignore untrusted events.
- browser_scroll gains a selector to scroll an element's container.

True async JS (async_eval.rs)
- callAsyncJavaScript awaits Promises — fixes click/type/wait/readiness
  which previously got an unserialised Promise (empty result).

Discovery
- browser_snapshot pierces open shadow DOM and surfaces listener-only
  clickables; header reports scroll position / page height; refs hidden
  behind a non-enumerable property.

New tools
- browser_console_messages / browser_network_requests (ring buffers).
- browser_handle_dialog / browser_upload_file (arm native answers; the
  open-panel delegate is replaced with a wry-fallback).
- Domain failures now surface as model-visible isError results.

Infra & tests
- OCTOWEB_MCP_PORT / OCTOWEB_CONFIG_DIR for isolated side-by-side instances.
- test_mcp.py + tests/fixtures: 28-test e2e suite (28/28 green, 3x),
  validated against Bing/GitHub. Docs updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Octomind — developer:brief (octohub:glm)

Now I have a comprehensive understanding of the changes. Let me compile the brief.

📦 Brief: MCP actionability overhaul, Liquid Glass UI, ACP session persistence, and keyboard focus fix

Overall risk: 🟡 MEDIUM · Cards: 5

# Change Risk Confidence
1 MCP interaction pipeline rewritten with actionability harness, async JS eval, and new tools (dialogs, uploads, console, network) 🟡 ●●●
2 ACP session persistence now saves tool runs and turn durations; SIGTERM shutdown prevents orphaned agents 🟡 ●●●
3 Keyboard focus restored to active webview after palette/address-bar closes 🟢 ●●●
4 Liquid Glass visual overhaul across all UI surfaces 🟢 ●●●
5 Config isolation (OCTOWEB_CONFIG_DIR), MCP port override, and user-tool symlinks 🟢 ●●●
Card 1/5: MCP interaction pipeline: actionability harness, async eval, dialogs/uploads, observability · 🟡 · ●●●

INTENT
Make MCP browser control reliable enough for autonomous AI agents: clicks actually land, Promises resolve, JS dialogs and file uploads can be answered programmatically, and page diagnostics (console, network) are observable.

WHAT CHANGED

  • All DOM interaction scripts (click, hover, type, press_key, select_option, scroll) extracted into dom_actions.rs with a shared Playwright-style actionability harness: resolve-with-retry (2.5s), bounding-box stability, occlusion detection, disabled/readonly gate, full pointer+mouse event sequences with composed: true for shadow DOM penetration
  • async_eval.rs replaces evaluate_script_with_callback with callAsyncJavaScript — Promises are now actually awaited instead of returning empty results
  • browser_navigate simplified: always opens background tabs (no new_tab/background flags); browser_switch_tab is the sole focus-changer. TabManager::open_background() added to keep is_active truthful
  • browser_switch_tab and browser_close_tab now mutate TabManager state synchronously before the event, so immediate get_tabs/get_current_tab calls see consistent state
  • New MCP tools: browser_handle_dialog (arm auto-answer for JS alert/confirm/prompt), browser_upload_file (arm file chooser with paths), browser_console_messages, browser_network_requests
  • browser_scroll gains selector parameter for scrolling element containers
  • browser_snapshot now pierces open shadow DOM, surfaces listener-only clickables (__octoweb_listeners), and reports viewport/scroll position in header
  • send_command return type changed from Result<T, McpError> to Result<Result<T, String>, McpError> — domain errors (tab gone, element missing) are now model-visible isError results via browser_try! macro instead of opaque protocol errors
  • dialog_patch.rs extended: JS dialogs auto-answered when armed via arm_dialogs(); file chooser intercepted via handle_open_panel (replaces wry's IMP, falls back to it when unarmed)
  • Observability ring buffers (__octoweb_console, __octoweb_net) injected by COMBINED_SCRIPT in webview_utils.rs — console log/error + fetch/XHR interception, capped at 200 entries
  • 28-test e2e suite (test_mcp.py) with local fixture HTTP server and 18 HTML fixtures

IMPACT RADIUS

  • All MCP tool consumers (AI agents) get richer error messages, reliable clicks, and two new diagnostic tools
  • send_command signature change is internal (not a public API break) but every tool handler was updated
  • dialog_patch.rs now replaces wry's runOpenPanelWithParameters: IMP at runtime — any future wry upgrade that changes this selector could silently lose the override

RISK
🟡 MEDIUM. The class_replaceMethod on wry's open-panel delegate is fragile across wry versions — if wry renames or removes that method, the original IMP stored in ORIG_OPEN_PANEL becomes a dangling pointer. The fallback (null check) prevents a crash but would silently disable file uploads. The actionability harness retries for 2.5s on missing elements, which is correct for late-rendering SPAs but could cause confusing delays on genuinely missing selectors. The __octoweb_listeners WeakSet tagging via addEventListener prototype patch is non-enumerable but still a prototype mutation — rare pages that seal or proxy EventTarget.prototype could throw.

DIVERGENCE
🧩 Incomplete change: browser_press_key in main.rs still has an inline script for the keypress dispatch (not using dom_actions::press_key_script), while click/hover/type/select_option were migrated. The press_key_script function exists in dom_actions.rs but isn't called from the MCP command handler yet.

📎 Source

Card 2/5: ACP tool-run persistence and SIGTERM shutdown · 🟡 · ●●●

INTENT
Agent tool runs (steps group) and turn durations now survive app restarts, and orphaned agent subprocesses are explicitly terminated on exit.

WHAT CHANGED

  • AcpSession gained tool_buf: Vec<AcpToolRecord>, tool_starts: HashMap<String, (usize, Instant)>, and turn_started: Option<Instant> — tool starts/updates are tracked in real time and flushed into the persisted agent message alongside agent_buf on Done/Cancel/Error/Disconnect
  • AcpMessage struct extended with tools: Vec<AcpToolRecord> and turn_ms: u64 — persisted to acp_history.json
  • flush_agent_turn() replaces the old push_acp_msg("agent", buf) pattern — drains agent_buf, tool_buf, and turn_started into one atomic message
  • Tool JSON payloads capped at 8KB (ACP_TOOL_JSON_CAP) to prevent history bloat from large MCP fetches
  • render_ui tool runs are excluded from persistence (they render as A2UI bubbles, persisted separately)
  • On Connected, if the agent's session ID differs from the saved one, the old ID is kept and a warning is shown — prevents silently orphaning the real conversation context
  • save_and_exit() now sends SIGTERM to all octomind subprocesses, waits up to 3 seconds, then SIGKILL stragglers — prevents orphan processes that would race --resume on next launch
  • ACP reconnect now passes --resume <saved_id> so agent context is restored instead of minted fresh

IMPACT RADIUS

  • acp_history.json format changes (new tools and turn_ms fields) — #[serde(default)] on both ensures backward compatibility with old files
  • All four flush sites (Done, Cancel, Error, Disconnect) unified through flush_agent_turn()

RISK
🟡 MEDIUM. The SIGTERM-then-SIGKILL in save_and_exit is correct but runs on the main thread with a 3-second blocking wait loop. If the app is quitting via std::process::exit (which this function does), this is fine — but if called from a different path, it blocks the UI. The tool_starts HashMap uses the tool-call ID as key; if an agent emits duplicate IDs (protocol violation), the second start would overwrite the first's index, leaving a stale entry in tool_buf.

📎 Source

  • src/main.rsflush_agent_turn, SIGTERM shutdown, --resume flag, tool_buf/tool_starts/turn_started fields
  • src/config.rsAcpToolRecord, AcpMessage.tools/turn_ms, base_dir()
Card 3/5: Keyboard focus restored to active webview after overlay dismissal · 🟢 · ●●●

INTENT
After closing the command palette or address bar (keyboard-driven surfaces), keystrokes no longer land nowhere — the page WebView regains first-responder status automatically.

WHAT CHANGED

  • focus_active_webview! macro added: checks is_app_active(), no overlay/settings/shortcuts visible, no sidebar key ownership, then calls browser_win.set_focus() + wv.focus() on the active tab's WebView
  • Called after: inline edit close, overlay close, address bar navigation completion, and system stats timer — all places where a secondary key window previously stole and never returned focus
  • Closing a background tab no longer triggers a tab switch (only closing the active tab does)

IMPACT RADIUS

  • Isolated to keyboard focus behavior after UI surface dismissal. No API or state changes.

RISK
🟢 LOW. The guard conditions (checking overlay, settings, sidebar visibility) prevent stealing focus from legitimate surfaces. wv.focus() calls makeFirstResponder: which is the standard AppKit pattern.

📎 Source

  • src/main.rsfocus_active_webview! macro and call sites
Card 4/5: Liquid Glass visual overhaul across all UI surfaces · 🟢 · ●●●

INTENT
Unify the browser's visual language around macOS Tahoe's "Liquid Glass" aesthetic: specular rim highlights, floating depth shadows, springy motion curves, and borderless card surfaces.

WHAT CHANGED

  • All *_html.rs modules updated: borders replaced with box-shadow: 0 0 0 0.5px outlines + inset 0 1px 0 specular highlights; border-radius increased (10→14-18px); animations switched to cubic-bezier(0.34, 1.56, 0.64, 1) spring curve; dark mode gets adjusted shadow opacity
  • Sidebar: glass capsule suggestion buttons, specular rim pseudo-elements on input row and tool groups, session tabs use transparent borders with accent glow shadows
  • New tab page, overlay, settings, shortcuts, find bar, notification toast, error page, quick slots — all received the same treatment
  • User message copy buttons now icon-only circles (text hidden) to avoid dwarfing small bubbles

IMPACT RADIUS

  • Visual only. No behavioral or API changes.

RISK
🟢 LOW. Pure CSS changes. The -webkit-mask-composite: xor technique for the specular rim is well-supported in WebKit/WKWebView.

📎 Source

Card 5/5: Config isolation, MCP port override, and user-tool symlinks · 🟢 · ●●●

INTENT
Enable side-by-side instances (daily browser + e2e test profile) and let agents discover user-installed tools.

WHAT CHANGED

  • config::base_dir() introduced: reads OCTOWEB_CONFIG_DIR env var, falls back to platform default. All config/session/history/favicon/prompt paths now route through it instead of calling dirs::config_dir() independently
  • crash_report.rs log dir also uses base_dir()
  • MCP server port configurable via OCTOWEB_MCP_PORT env var (default 3434)
  • a2ui_render_ui::install() now symlinks ~/.agents/tools/* into the workspace tools directory (skipping render_ui itself and existing entries), and cleans up dangling symlinks

IMPACT RADIUS

  • E2e tests can run isolated profiles without conflicting with the daily browser
  • User tools are discoverable by agents without manual PATH configuration

RISK
🟢 LOW. OCTOWEB_CONFIG_DIR and OCTOWEB_MCP_PORT are opt-in env vars with sensible defaults. The symlink logic is append-only and skips existing entries.

📎 Source

📂 Files changed (46 files, ~4750 lines)

{tokens:460407,cost:0.4773272}

- Consolidate multi-line PathBuf joins into single lines
- Standardize brace placement in render loop
- Reformat browser_try macro calls for better readability
- Clean up whitespace in log directory path construction
@donk8r
donk8r merged commit e526450 into master Jun 15, 2026
@github-actions

Copy link
Copy Markdown

Octomind — developer:brief (octohub:glm)

Confirmed — all changes are formatting-only. Here's the brief:

📦 Brief: Style-only formatting cleanup across 4 files

Overall risk: 🟢 LOW · Cards: 1

# Change Risk Confidence
1 Reformat browser_try! macro calls and PathBuf joins for readability 🟢 ●●●
Card 1/1: Reformat `browser_try!` macro calls and `PathBuf` joins for readability · 🟢 · ●●●

INTENT
Make multi-line expressions easier to read by standardizing formatting — no behavioral change.

WHAT CHANGED

  • All 20 browser_try!(…) invocations in mcp.rs reformatted: the macro call and its inner send_command(…).await? now sit on separate lines with consistent indentation
  • Seven PathBuf chain expressions in config.rs collapsed from multi-line .join("octoweb").join("…") into single-line form
  • crash_report.rs log_dir() similarly collapsed
  • a2ui_render_ui.rs let … else { continue } block reformatted to multi-line style

IMPACT RADIUS
Isolated change — no downstream impact detected. The browser_try! macro definition itself is unchanged (match $e { Ok(v) => v, Err(msg) => … }), so all call sites produce identical AST output. Path expressions evaluate to the same PathBuf values.

RISK
🟢 LOW. Pure whitespace/reformatting — zero logic change. Any of these would compile to identical machine code.

📎 Source

📂 Files changed (4 files, ~242 lines)

{tokens:256945,cost:0.2608698}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant