Office World Actions: chat-commanded errands in the 3D world

lat.md/lat.md

@@ -25,4 +25,5 @@ This directory defines the high-level concepts, business logic, and architecture
 - [[office-3d-interiors]] — enterable office/bank/showroom interiors: per-location conditional mounting (city unmounts while indoors), camera fly-in rig, interactable objects (ATM → wallet, desk → agent, car → spec card), and idle-agent walking trips between buildings.
 - [[office-3d-walk-mode]] — GTA-style walk mode: glass roofs over the enterable buildings, the user's own third-person avatar (WASD + chase camera, shared crowd/collision), doorway-driven interior loading, and proximity Press-E interactions.
 - [[office-interactions]] — space representatives: interactive bank tellers whose menu runs account status, balances, and account creation against the hermes-one backend for a chosen agent; the extensible pattern for future spaces (showroom sales, building space).
+- [[office-world-actions]] — chat-commanded errands: the agent's LLM emits world-action blocks from the office chat, its avatar walks the trip route to the bank/showroom, and the rep modal auto-opens running the requested action on arrival.
 - [[mcp-servers]] — add / edit / remove / enable / test MCP servers from the Capabilities → MCP tab; the shared add+edit modal and the in-place atomic `updateMcpServer` upsert (config.yaml locally, gateway REST in Remote/SSH).

lat.md/office-3d-interiors.md

@@ -56,6 +56,8 @@ Everything lives in [[src/renderer/src/screens/Office/office3d/core/collision.ts
 
 Blocked walkers wall-follow: when the push-out cancels a step, the walker commits to a short slide along the blocking face's tangent — signed toward its goal, re-derived at corners — with goal steering suspended for the burst. (A per-frame perpendicular nudge is not enough: the goal pull re-pins the walker against the face each frame, walking in place forever — the original stuck-at-desk bug.) The static resolver exposes its push normal for this via `resolveStaticColliders`' `pushOut`.
 
+Wall-follow handles convex obstacles only — in a concave pocket (a wall corner plus furniture) the two faces re-derive opposite slides every frame and the walker deadlocks in place. Doorways are therefore routed explicitly: [[src/renderer/src/screens/Office/office3d/core/routing.ts#routeTarget]] makes every room crossing a two-hop "door gate" (near-side gate point first, then the far side), so the wall-crossing hop passes through the door gap from any start position. The CEO office's own doorway is routed before the divider — a CEO→east walk that aims at the partition door first pins against the glass (the stuck-in-the-corner bug, exposed when [[office-world-actions|chat-commanded missions]] started walks from arbitrary desks). [[src/renderer/src/screens/Office/office3d/core/routing.test.ts]] walks the previously-deadlocking routes hop by hop and asserts no hop segment crosses an interior wall.
+
 Desk seats add a structural rule: agents approach the chair from the open side — up the desk-free aisle between desk columns, then across at seat height (`deskApproachByAgent` in AgentsLayer) — so the everyday sit-down never depends on obstacle avoidance at all. Trip/NPC waypoints sit clear of all colliders with looser arrival radii so a crowded waypoint can't strand anyone. Pedestrians register in the same crowd as visiting agents, so the two populations avoid each other too; standing staff participate as static circles.
 
 ## Agent trips
@@ -64,6 +66,6 @@ Idle agents occasionally walk out of the office to the bank or showroom, wander
 
 The canvas↔world mapping ([[src/renderer/src/screens/Office/office3d/core/geometry.ts#worldToCanvas]]) is linear, so waypoints far outside the office's 0..1800 rectangle work unchanged — no second coordinate system.
 
-The controller in [[src/renderer/src/screens/Office/office3d/objects/AgentsLayer.tsx#AgentsLayer]] adds a "trip" mode (phases out → wander → back) beside toSeat/seated. Only idle (non-working) seated agents start trips, capped at `TRIP_MAX_TRAVELLERS`; if an agent's gateway starts mid-trip it walks the route home in reverse rather than teleporting. Each agent's `place` ("office" | "bank" | "showroom" | "outside") is derived from route progress.
+The controller in [[src/renderer/src/screens/Office/office3d/objects/AgentsLayer.tsx#AgentsLayer]] adds a "trip" mode (phases out → wander → back) beside toSeat/seated. Only idle (non-working) seated agents start trips, capped at `TRIP_MAX_TRAVELLERS`; if an agent's gateway starts mid-trip it walks the route home in reverse rather than teleporting. Each agent's `place` ("office" | "bank" | "showroom" | "outside") is derived from route progress. Trips can also be commanded from the office chat — a mission replaces the wander with a "visit" at a named interaction stop; see [[office-world-actions#Office World Actions#Commanded trips]].
 
 The simulation always runs for every agent; the `visiblePlace` prop only toggles per-agent wrapper-group visibility each frame, so each interior view shows exactly the agents actually in that building and the city view shows everyone, including walkers on the street.

lat.md/office-interactions.md

@@ -2,7 +2,7 @@
 
 Interactables in the Office tab's bank that agents do business with. Clicking the bank teller or the self-service ATM opens a shared action modal (balances, account status, and more) run against the hermes-one backend for the chosen agent.
 
-The bank is the first transaction space; future spaces (car showroom sales, building space) reuse the same pieces: a registry entry, an [[src/renderer/src/screens/Office/office3d/objects/Interactable.tsx#Interactable]] hookup in the interior, and action wiring in the panel. Coming-soon actions (the ATM's withdraw/deposit) are registered with a `disabled` flag and render as muted "Soon" chips until their flows land.
+The bank is the first transaction space; future spaces (car showroom sales, building space) reuse the same pieces: a registry entry, an [[src/renderer/src/screens/Office/office3d/objects/Interactable.tsx#Interactable]] hookup in the interior, and action wiring in the panel. Coming-soon actions (the ATM's withdraw/deposit) are registered with a `disabled` flag and render as muted "Soon" chips until their flows land. Beyond clicking, the panel can also be opened by an agent itself: a chat-commanded errand walks the agent to the rep and auto-opens the panel with the requested action — see [[office-world-actions]].
 
 ## Representative Registry
 
@@ -22,6 +22,8 @@ A centered modal shared by every representative (bank teller, ATM): a dimmed bac
 
 [[src/renderer/src/screens/Office/RepInteractionPanel.tsx#RepInteractionPanel]] takes the rep, the agent list, and an initial agent. Office passes `selectedId ?? defaultAgentId`, where `defaultAgentId` is the active profile (the `profile` prop, matched by agent id) or the first agent — so the picker opens pre-selected on the current profile instead of an empty "Choose an agent…". It renders a fixed inset-0 backdrop (dismiss on backdrop click or Escape) centering the card; the card fades/scales in on mount and its layout is self-contained inline styles (not Tailwind classes) so positioning can't break. The header identity is per-representative: a `REP_ICONS` glyph (bank landmark for the teller, card for the ATM), the rep's name, a live status line with a green dot (teller "Open · serving {agent}", ATM "Online · {agent}") and an agent-picker chip (an avatar + native `<select>` so it stays keyboard-accessible). A hero card morphs with the latest action outcome — total-balance figure with a decorative flat sparkline ([[src/renderer/src/screens/Office/RepInteractionPanel.tsx#BalanceSparkline]], carries no data since the backend has no price history), a loading skeleton, a success card (address · Base), an error card with a **Retry** button that re-runs the last action, a warning hint, or the dedicated empty-wallet card (`$0.00`, "no funds in this account yet" — no create prompt, since reaching a balance means the account exists) when a loaded balance is zero. Below it, token rows (round symbol badge, name, amount, USD) render on a loaded balance and account-status rows render transactable/receive-only badges. Money figures (the total, token amounts, and USD values) render in Space Grotesk via the `--font-numeric` token — a self-hosted `@font-face` in `src/renderer/src/assets/main.css` (`fonts/SpaceGrotesk-Variable.ttf`), no CDN.
 
+A mission-driven open (see [[office-world-actions#Office World Actions#Orchestration]]) passes `autoAction`, which the panel runs exactly once after the account scope resolves — the `accountResolved` gate exists because the accountId resolution changes the cache key, bumps `requestSeq`, and would drop an action fired at mount.
+
 Actions render as flex-wrapping chips (`visibleActions`) rather than a fixed grid, so a variable set always lays out cleanly with no empty cells: **check balance** (primary/accent chip) reads the cached transactable wallet id when present — otherwise looks it up via `syncWallets` — and renders its backend portfolio; **account status** lists the linked cloud agent's wallets via `syncWallets`; **create account** (teller only) provisions a backend wallet, mapping the 409 "already provisioned" reply to a friendly notice. Disabled actions — the ATM's **withdraw** and **deposit** — render as muted chips with a "Soon" badge. Because an agent has exactly one account, **create account is dropped once the agent is known to have one** ([[src/renderer/src/screens/Office/RepInteractionPanel.tsx#rememberBank]]'s `hasAccount`). Signed-out, unlinked, and foreign states (the agent's cloud link belongs to a different Hermes account) render hints instead of errors. Results are guarded by a per-request token: switching the agent picker invalidates any in-flight action, so a late response can never render one agent's wallets under another.
 
 ### Session cache

lat.md/office-world-actions.md

@@ -0,0 +1,62 @@
+# Office World Actions
+
+Chat-commanded errands in the Office 3D world: tell an agent "go to the bank and check my balance" and its avatar walks the trip route to the bank, stops at the teller or ATM, and the interaction modal opens with the requested action already running.
+
+The agent's own LLM does the understanding — the chat injects a vocabulary of world abilities, the model emits a machine-readable action block alongside its natural reply, and the renderer executes it. Adding a future ability (transfers, car purchase, other rooms) is one vocabulary entry plus a handler, not a new pipeline.
+
+The pipeline has four stages, deliberately decoupled: protocol ([[src/renderer/src/screens/Office/office3d/interactions/worldActions.ts]]) → orchestration ([[src/renderer/src/screens/Office/Office.tsx]]) → bus ([[src/renderer/src/screens/Office/office3d/interactions/missionBus.ts]]) → simulation ([[src/renderer/src/screens/Office/office3d/objects/AgentsLayer.tsx#AgentsLayer]]).
+
+## Action protocol
+
+The office chat prepends a request-side system message describing the abilities; when the user's request maps to one, the model appends a fenced ```world-action JSON block to its reply, which the renderer strips from the visible text and validates.
+
+[[src/renderer/src/screens/Office/office3d/interactions/worldActions.ts#buildWorldActionSystemPrompt]] generates the vocabulary from a declarative `ABILITIES` list (one entry per ability, so prompt and parser can't drift apart). Abilities today: `go_to` (bank, showroom) and `bank` operations (check_balance / account_status / create_account, via teller or ATM — create_account normalises to the teller, its only flow). [[src/renderer/src/screens/Office/office3d/interactions/worldActions.ts#parseWorldActions]] is tolerant by design: malformed JSON and unknown `do` values are stripped from the text but run nothing, so a newer prompt never breaks an older desktop. [[src/renderer/src/screens/Office/office3d/interactions/worldActions.ts#planWorldActions]] reduces a validated action list to one plan — a walking destination plus at most one facility interaction; bank operations force the bank as destination.
+
+The system message is injected into `sendMessage`'s history array (role "system") in [[src/renderer/src/screens/Office/OneChatModal.tsx]], so it is never persisted; transcripts stay clean and reloads only strip blocks (via `toChatMessages`) without re-running them — a reopened chat can never replay an old errand.
+
+## Mission bus
+
+A tiny module-level pub/sub bridging the DOM side (chat, modals, camera) and the r3f Canvas simulation, which mutates refs per-frame and can't share React context across the Canvas boundary.
+
+A `Mission` is one commanded errand: agent id, destination, and an optional interaction (rep id + panel action). [[src/renderer/src/screens/Office/office3d/interactions/missionBus.ts#dispatchMission]] hands it to the simulation, [[src/renderer/src/screens/Office/office3d/interactions/missionBus.ts#completeMission]] tells the sim the modal closed, and [[src/renderer/src/screens/Office/office3d/interactions/missionBus.ts#emitMissionEvent]] reports progress back ("arrived", "ended"). Plain Sets, no React, no allocations in the frame path — the low-end-device rule for all of Office 3D.
+
+## Commanded trips
+
+A mission reuses the existing trip machinery ([[office-3d-interiors#Office 3D Interiors#Agent trips]]) — same routes, collision, crowd separation — with a "visit" phase instead of random wandering.
+
+On dispatch, [[src/renderer/src/screens/Office/office3d/objects/AgentsLayer.tsx#AgentsLayer]] puts the agent's controller into trip mode on [[src/renderer/src/screens/Office/office3d/trips.ts#getTripRoute]], joining at [[src/renderer/src/screens/Office/office3d/trips.ts#nearestRouteIdx]] so a mission can start from a desk, the rest room, or mid-trip anywhere; an agent already inside the destination skips straight to the visit. Office-interior legs route through the partition/CEO doorways via [[src/renderer/src/screens/Office/office3d/core/routing.ts#routeTarget]]'s door gates (missions can start at any desk — random trips only ever started in the rest room, which is why single-hop routing used to suffice; see [[office-3d-interiors#Office 3D Interiors#Collision]]). Unlike random trips, a working agent is not turned around: being sent on an errand while working is the point.
+
+The visit phase walks to the interaction's named stop — `TripRoute.stops`, keyed by rep id, reusing proven collision-clear wander points (teller counter, ATM row) — emits "arrived", then stands holding until `completeMission` or a timeout (2 min with a modal pending, 15 s for a plain go_to), and finally walks the route home in reverse. A superseding mission for the same agent ends the previous one first.
+
+## Orchestration
+
+Office.tsx owns the mission lifecycle: it turns parsed actions into a mission, points the camera, and opens the interaction modal on arrival.
+
+`handleWorldActions` (in [[src/renderer/src/screens/Office/Office.tsx]]) plans the actions, dispatches the mission, closes the chat, and pulls back to the city view to watch the walk (walk mode keeps its own camera). On "arrived" it selects the agent, flies into the destination interior, and — when the mission has an interaction — opens [[src/renderer/src/screens/Office/RepInteractionPanel.tsx#RepInteractionPanel]] with an `autoAction`, which the panel runs exactly once after its account scope resolves (running earlier would race the cache-key change that bumps `requestSeq` and drops in-flight results). Whichever way the modal goes away (close, Escape, exiting the interior, walk-mode movement), an effect watching `activeRepId` completes the mission so the agent heads home. The reverse direction holds too: if the simulation times out the interaction hold and emits "ended" while the panel is still open, the panel is closed with it — UI and mission state always describe the same interaction.
+
+## Tests
+
+Vitest suites covering the protocol and the bus; the walking behaviour rides on the existing trip machinery covered by the interiors docs.
+
+- [[src/renderer/src/screens/Office/office3d/interactions/worldActions.test.ts]] — prompt/parse/plan
+- [[src/renderer/src/screens/Office/office3d/interactions/missionBus.test.ts]] — pub/sub contract
+
+### Prompt advertises every ability
+
+Every JSON example the vocabulary prompt shows must itself parse as a valid action — a vocabulary the parser rejects would teach the model dead syntax.
+
+### Parses and strips action blocks
+
+A reply carrying a world-action block yields the validated actions and clean display text; a bare object is accepted as a one-item array.
+
+### Tolerates malformed blocks
+
+Malformed JSON and unknown abilities are stripped from the visible text but produce no actions — bad model output degrades to a normal chat reply, never a crash or a wrong errand.
+
+### Bank operations force the bank
+
+Planning maps a bank operation to the bank destination and the right representative (teller vs ATM), overriding any conflicting go_to, and a plain go_to yields a walk with no interaction.
+
+### Bus delivers and unsubscribes
+
+Missions, completions, and events reach subscribers; unsubscribed listeners stop receiving so a torn-down AgentsLayer never acts on a mission with a stale controller map.