Office World Actions: chat-commanded errands in the 3D world

[fathah]

Tell an agent in the office chat "go to the bank and check my balance" — its avatar stands up, walks the sidewalk route to the bank, stops in front of the teller (or the ATM if you asked for it), the camera flies in after it, and the interaction modal opens with the balance already fetching. Closing the modal sends the agent walking home.

The agent's own LLM does the understanding — no keyword matching. This is the first step toward a fully intelligent 3D world (agentic transactions, purchases, driving, more rooms), and the pipeline is built so each future ability is a small extension, not a new system.

How it works

Four decoupled stages:

OneChatModal ──parse──▶ Office.tsx ──mission──▶ missionBus ──▶ AgentsLayer (r3f sim)
 (protocol)          (orchestration)            (pub/sub)      (visit trips)

1. Protocol — office3d/interactions/worldActions.ts
   The chat injects a vocabulary of world abilities as a request-side system message (never persisted — transcripts stay clean). When the user's request maps to an ability, the model appends a fenced ```world-action JSON block to its reply. The renderer strips the block from the visible text and validates it.

   • The vocabulary prompt is generated from a declarative ABILITIES list, so prompt and parser can't drift apart. Adding "send 0.1 ETH to prompt-engineer" later = one catalog entry + a parse case + a plan case.
   • Tolerant by design: malformed JSON and unknown do values are stripped but run nothing — bad model output degrades to a normal chat reply, never a wrong errand.
   • World actions ship only in the fresh reply; reloaded transcripts are merely stripped, so reopening a chat can never replay an old errand.

2. Orchestration — Office.tsx
   Plans the parsed actions into a mission (bank operations force the bank; teller vs ATM picks the representative), dispatches it, closes the chat, and pulls the camera back to the city so you watch the walk. On arrival: fly into the building, select the agent, open RepInteractionPanel with an autoAction that runs as if clicked. Any way the modal goes away (close, Escape, exiting the interior, walk-mode movement) completes the mission and sends the agent home. Walk mode keeps its own camera — no hijacking.

3. Mission bus — office3d/interactions/missionBus.ts
   A tiny module-level pub/sub bridging the DOM UI and the r3f Canvas (React context can't cross the Canvas boundary, and the sim mutates refs without re-rendering). Plain Sets, zero allocations in the frame path — consistent with the low-end-device efficiency contract of the rest of Office 3D.

4. Simulation — AgentsLayer.tsx + trips.ts
   Missions reuse the existing trip machinery (routes, collision, crowd separation) with a new "visit" phase: join the route at the nearest waypoint (missions can start from any desk, the rest room, or mid-trip), walk to a named interaction stop (TripRoute.stops, keyed by rep id — reusing proven collision-clear points at the teller counter and ATM row), emit arrived, hold while the modal is open (2 min cap; 15 s for a plain "go there"), then walk home in reverse. Unlike random trips, a working agent is not turned around — being sent on an errand while working is the point. A superseding mission cleanly ends the previous one.

Bug fix: door-gate routing (stuck-agent deadlock)

Commanded missions exposed a latent routing bug: routeTarget checked the partition crossing before the CEO-office crossing, and aimed at the far side of a door in one hop. From a CEO seat (or anywhere south of the door band) the straight line crossed a wall outside the door gap, and wall-follow deadlocks in concave pockets (wall corner + couch/plants) — the agent walked in place forever. Random trips never hit this because they only ever started from the rest room.

Fixed in a new shared module office3d/core/routing.ts: every room crossing is a two-hop door gate (near-side gate point first, then the far side), so the wall-crossing hop passes through the door gap from any start position, and the CEO office's own doorway is routed before the divider. Regular seat walks get the fix too (the CEO → rest-room walk had the same latent bug).

RepInteractionPanel: autoAction

New optional prop that runs a commanded action exactly once when the panel opens — gated on the account scope having resolved, because the accountId resolution changes the cache key, bumps requestSeq, and would silently drop an action fired at mount.

Tests

• worldActions.test.ts — every advertised prompt example must itself parse (vocabulary can't teach dead syntax); parse/strip round-trip; malformed and unknown blocks degrade safely; create_account normalises to the teller; planning forces the bank and picks the right rep.
• missionBus.test.ts — pub/sub contract incl. unsubscribe (a torn-down AgentsLayer must never act on a stale mission).
• routing.test.ts — replays the previously-deadlocking routes hop by hop against the real wall rectangles from layout.ts and asserts no hop crosses a wall; verified to fail against the old logic.

Full suite: 1770 passing (166 files), typecheck + lint clean, lat check passes.

Docs

New lat.md/office-world-actions.md (protocol, bus, commanded trips, orchestration, test specs) plus cross-links from office-interactions.md and office-3d-interiors.md (door-gate routing under Collision).

Extending (the recipe)

To add a future ability (transfer, purchase, drive, meeting room):

1. Add an ABILITIES entry in worldActions.ts (the prompt updates itself) + a parseAction case + a planWorldActions case.
2. If it's a new destination: a TripRoute (+ named stops) in trips.ts.
3. If it's a new facility interaction: a registry entry + panel wiring (existing space-representative pattern).

Notes / follow-ups

• The walk-out phase has no stuck-timeout; the two known wall-crossing routes are fixed and regression-tested, but a generic "no progress → abandon errand" failsafe is a good future hardening.
• The dashboard-WS transport maps the injected system history role to user (apiHistory); the instruction still works, but a first-class system-message field there would be cleaner.
• A broader Office 3D performance audit (draw calls, material sharing, shadow cost, low-quality toggle) is tracked separately toward the low-end-device goal.

[greptile-apps]

Greptile Summary

This PR adds chat-commanded errands to the Office 3D world. The main changes are:

• A model-driven world-action protocol for chat replies.
• Mission orchestration between chat, the UI, and the 3D simulation.
• Agent visits with destination stops, interactions, and return trips.
• Automatic representative actions after account resolution.
• Door-gate routing for reliable room crossings.
• Tests and documentation for the new workflow.

Confidence Score: 5/5

This looks safe to merge.

• CRLF world-action blocks are now parsed and stripped correctly.
• Timed-out missions now close their representative panels without completing stale mission state.
• No blocking issues were found in the updated code.

Important Files Changed

Filename	Overview
src/renderer/src/screens/Office/office3d/interactions/worldActions.ts	Defines the world-action protocol and now accepts CRLF opening fences.
src/renderer/src/screens/Office/Office.tsx	Coordinates mission dispatch, arrival, representative panels, camera state, and mission completion.
src/renderer/src/screens/Office/OneChatModal.tsx	Injects the action vocabulary, parses fresh replies, and strips protocol blocks from chat text.
src/renderer/src/screens/Office/RepInteractionPanel.tsx	Runs a mission-requested representative action after the account scope resolves.
src/renderer/src/screens/Office/office3d/objects/AgentsLayer.tsx	Adds commanded visit phases, mission events, hold behavior, supersession, and return trips.
src/renderer/src/screens/Office/office3d/core/routing.ts	Adds two-hop doorway routing for room and CEO-office crossings.

_{Reviews (3): Last reviewed commit: "Fix Greptile review findings on world ac..." | Re-trigger Greptile}