diff --git a/.env.example b/.env.example index 835a866..17c7bd6 100644 --- a/.env.example +++ b/.env.example @@ -28,7 +28,7 @@ HEYGEN_API_KEY= # - chat_ingester (Haiku ack writer pre-bakes per-row ack text) # Same key for both — the showrunner uses Sonnet 4.6, chat_ingester # uses Haiku 4.5; both consume from the same account. -ANTHROPIC_API_KEY=sk-ant-... +ANTHROPIC_API_KEY= # ─── Optional: GitHub (richer repo fetch) ──────────────────────────── @@ -54,7 +54,7 @@ OBS_HEARTBEAT_PASSWORD= # # Tokens expire ~60 days; regenerate when you see # `twitch_reader.auth_failed` in chat_ingester logs. -TWITCH_BOT_TOKEN=oauth:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx +TWITCH_BOT_TOKEN=oauth: # The bot account's Twitch login (lowercase). Create a second Twitch # account or reuse one you own — this is who the token belongs to. @@ -141,17 +141,17 @@ CHAT_QUEUE_DB=state/chat_queue.sqlite # If no sink is configured, alerts just log at WARNING. # Slack-compatible incoming webhook URL. -# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/... +# SLACK_WEBHOOK_URL= # Discord-compatible incoming webhook URL. -# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... +# DISCORD_WEBHOOK_URL= # OBS heartbeat alerts (separate stdlib-only script under scripts/). # Point this at the SAME Slack/Discord webhook as above to consolidate # all alerts into one channel. The OBS heartbeat doesn't share the # Notifier rate-limit — set this to the same URL and you'll see OBS # alerts alongside CostCap + failure_alarm alerts. -# OBS_HEARTBEAT_ALERT_WEBHOOK=https://hooks.slack.com/services/... +# OBS_HEARTBEAT_ALERT_WEBHOOK= # macOS-native notification (osascript). '1' / 'true' / 'yes' enables; # anything else (or unset) disables. Useful for local development; diff --git a/.gitignore b/.gitignore index b1acbc5..3cd8542 100644 --- a/.gitignore +++ b/.gitignore @@ -43,6 +43,10 @@ Thumbs.db *.swp *~ +# Assistant-local workspace metadata +.claude/ +.codex/ + .browser-profile/ -logs/ \ No newline at end of file +logs/ diff --git a/.mcp.json b/.mcp.json new file mode 100644 index 0000000..4d80852 --- /dev/null +++ b/.mcp.json @@ -0,0 +1,8 @@ +{ + "mcpServers": { + "temporal-docs": { + "type": "http", + "url": "https://temporal.mcp.kapa.ai" + } + } +} diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b1b41bb --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,687 @@ +# Live Streamer Changelog + +## 2026-05-28 + +### Implemented (across multiple sub-sessions) + +**Episode v2 + §7+§8 transition + autonomous discovery (Plans 1-7 + transition spec)** +- Rich capture pipeline (`episode_capture.py`) — one hidden-browser snapshot per episode produces an `EvidenceBundle` (clean README + document-ordered outline + anchors + activity sidebar). +- Single-call episode author (`episode_author.py`) — one LLM call → ordered scenes + cue intents + verdict, both streaming and one-shot variants. `SHOW_VOICE` extracted into its own module; calibration ladder + bouncer persona. +- Deterministic grounder (`episode_grounder.py`) — cue intents → dense cues with the point-first auto-scroll model; visit_link cue grounded as deterministic click+back pair. +- Runtime wiring — `produce_episode` + `produce_episode_stream` chained into `autonomous_loop`; reactive fallback unchanged. +- Chat seeding (Module 4 → showrunner) — `chat_request_queue` consumed by `_select_next_episode_source` with phase-A walk + collapsed-callback prepending. +- Cross-episode memory (`ShowLedger`) — last 6 episode summaries rendered into the next author's `memory=` payload (Plan 6). +- Verdict persistence (`verdict_store.py` + leaderboard overlay) — per-repo verdicts written to `state/verdicts.sqlite` on verdict-scene serve, read by the existing `/overlay/leaderboard_data` endpoint (Plan 7). +- §7+§8 transition layer (`episode_transitions.py`) — four `TransitionKind`s (DIRECT_ACK / GOOGLE_ACK / CALLBACK / TRENDING_BROWSE), each with its own one-shot author prompt and template fallback. Autonomous trending picks rank-1 fresh repo (filtered against `ShowLedger.recent_slugs()`). +- Link curiosity (`link_peek.py`) — `httpx` pre-fetch on outline links matching `CURIOSITY_KEYWORDS`; up to 8 peeks per bundle. `visit_link` cue lets the avatar visit a demo/docs/playground page mid-scene and return. + +**Cross-session anti-repeat (Plan 8 — this session)** +- `VerdictStore.recent_summaries(n=6)` reads MAX(id)-per-repo for boot hydrate. +- `ShowLedger.hydrate(summaries)` — idempotent bulk-append, preserves newest-first order. +- `AutonomousLoop` gains `show_ledger=` kwarg; runner builds the hydrated ledger between `VerdictStore.open()` and the loop construction. A 6 AM bounce no longer re-roasts yesterday's repos. + +**Notifier → failure alarms** +- New `_dispatch_alarm_alert` helper on `AutonomousLoop`; `planner_failure_alarm` and `browser_failure_alarm` sites now also send structured alerts through the shared `Notifier` instance (Slack / Discord / macOS, rate-limited). + +**CostCap wiring** +- `AnthropicSceneClient` accepts `cost_cap=`; `complete()` and `complete_stream()` gate on `should_block()` before HTTP and record `usage.input_tokens + output_tokens` after success. A blocked call raises `ScenePlannerError("cost_cap_blocked")`, the failover planner takes over. +- Runner builds `CostCap.from_env() + Notifier.from_env()` once at boot and shares the instances across reactive, episode-author, and transition LLM clients. + +**Bug fixes** +- Transition flag lifecycle (`_episode_in_transitions`): no longer cleared in the stream's `finally:` block on success — must wait until visible URL matches `_episode_repo_url` (or a 90s safety timeout). Stops the cascade where the LLM finishes streaming ~30s into the avatar's ~90s narration and the divergence guard dropped the pick + roast scenes (verified live in the 08:01 run: 9 dropped-N events totalling 39 scenes). +- Verdict beat exempt from `EPISODE_MAX_ROAST_SCENES` cap (the verdict scene IS the spoken score; capping it drops the audio). +- TRENDING_BROWSE navigate cue anchored EARLY in the narration (`_find_early_anchor`) — late-anchored cues miss when word_tracker hits prefix_timeout. Mitigation for the freeze cascade. +- `_find_late_anchor` fallback returns ONE word not joined two — fixes the "HTML Sure" anchor-not-in-narration bug that dropped browse segments. +- Beat-uniqueness guard in both `author_episode` and `author_episode_stream` — drops a second `install` scene (verdict beat exempt). +- Trending-opener anti-repeat: banned-phrase list in the prompt + soft-warn observer (`episode.transition_banned_opener`). +- Per-cue-intent drop telemetry in the grounder (`episode.cue_intent_dropped reason=…`) — turns the silent `under_grounded` 0-cue scene failure mode into something grep-able. +- Visit_link: prompt rule relaxed from "≤1 per episode" to "1-2 per episode" with broader curiosity-keyword pool (27 keywords up from 11; cap raised 5→8). + +**Operations** +- `.env.example` — durable template documenting required `ANTHROPIC_API_KEY` + Twitch creds; opt-in vars for episode mode, chat queue, verdict store, cost cap, notifier, OBS heartbeat consolidation. **Loudly documents** that chat consumption requires `SHOWRUNNER_EPISODE_MODE=true` + `CHAT_QUEUE_DB=...` (the bot will write rows but the avatar will ignore them without both). +- `docs/operations/mac-mini-setup.md` — full unattended Mac Mini setup for 7-day operation: system settings, OBS scene composition, launchd plists (5 — M1, showrunner, chat ingester, OBS heartbeat, disk watchdog), Slack/Discord webhook wiring. +- New design specs in [docs/superpowers/specs/](docs/superpowers/specs/) — episode v2, transition+discovery, Plan 8 cross-session anti-repeat (executed), Plan 9 chat-pulse + @-mentions (deferred). + +### Verified + +- `showrunner`: full suite → **896 passing, 7 deselected** (opt-in live tests). +- `stream_orchestrator`: full suite → **167 passing**. +- `chat_ingester`: full suite → **87 passing**. +- `script_producer`: legacy timing-library subset → **47 passing**. +- **Total: 1,197 tests, all green.** + +### Open / deferred + +- Plan 9 chat-pulse + @-mention path — spec only, no code. +- `word_tracker.prefix_timeout` — surfaces under load. Showrunner-side mitigation (early-anchor navigate cue) is in; deeper fix is in the legacy `script_producer/word_tracker.py`. +- OBS heartbeat → `showrunner.notify` unification — documented in `.env.example` (point both webhook envs at same URL). +- Long-soak live verification (≥1 hour) on the Mac Mini setup. + +--- + +## 2026-05-25 + +### Implemented + +- Fixed headed live-tool layout so `run_autonomous_loop --headed --fullscreen` uses the requested OBS page viewport by default instead of inheriting macOS fullscreen's smaller native viewport. + - Default headed layout is now fixed `obs-1080p` page viewport. + - `--native-viewport` is the explicit escape hatch for real native viewport probing. + - Added `showrunner/tools/probe_browser_layout.py` for screenshot/viewport verification. +- Tightened LLM-authored scene budget from `43-50` words to a safer `42-49` planner target while keeping the hard `17-20s` segment contract at `2.5 words/sec` with a small estimate tolerance near the lower bound. +- Added pre-push planner repair for "too wordy but not technically over 20s" authored scenes, so `50+` word scenes get repaired before they reach HeyGen. +- Hardened visible target validation: + - `text=...` highlight/point/click/type cues are rejected during planning when the current viewport snapshot has elements and the target is not visible. + - `element:N` targets still resolve through the snapshot. + - visible substring text targets such as `text=Browser automation` are allowed when the snapshot contains `Browser automation headline`. +- Added regression coverage for the real failure shape where `text=clangd-lsp` existed somewhere in the DOM but had no visible box in the current viewport. +- Replaced stream-facing fallback narration that said "the planner tripped" with natural host language. Internal fallback reasons are now logged on fallback use and persisted on segment completion. +- Changed the live autonomous default to `max_in_flight_scenes=2` for smoother speech continuity. Operators can still force the conservative mode with `MAX_IN_FLIGHT_SCENES=1`. +- Added a safety guard for `max_in_flight_scenes=2`: unresolved page-changing handoffs block the next push until their browser cue is observed, so the avatar does not critique a hidden/preflight page before the visible browser has actually moved. +- Tightened the autonomous planner personality and repo-review contract: + - host voice is now specified as sharp late-night software critique with visible receipts, not neutral product narration, + - GitHub repo scenes must prioritize README/product substance, docs, install/quickstart, examples, homepage/demo links, and repair notes over stars, branches, tags, commits, sponsor buttons, nav tabs, and file-tree trivia, + - visible links now force visible `click` cues instead of lazy direct `navigate` cues when the destination is already on screen. +- Reweighted browser scout target extraction so GitHub README/docs/about/homepage/example/install/demo content outranks global GitHub chrome and metadata. +- Reset Chrome page zoom on browser start and after page changes/new tabs/searches to avoid persistent-profile zoom making the OBS browser look shifted or cropped. +- Added a two-pass scene repair ceiling for chained planner errors. Example: length repair can reveal metadata chatter, and the second repair now gives a stricter rewrite-from-scratch instruction instead of failing immediately. +- Narrowed the GitHub metadata gate so `tags` metadata is blocked, but normal wording such as "anchor tag" is allowed. +- Ran a real prompt-iteration pass against the autonomous planner contract: + - eval reports now count cue execution failures as failed turns instead of merely recording them, + - eval summaries include cue op histograms, scroll counts, and executed-cue success rate, + - planner payloads explicitly tell Claude that screenshots are visual evidence but image/screenshot text may not be selectable, + - director rejects PDF-like `navigate`, `navigate_address_bar`, and `new_tab` targets so arXiv PDFs do not trigger downloads in the stream browser. +- Rejected one prompt experiment after measurement: a soft "make it more human" anti-static instruction improved neither motion nor reliability and caused two real-turn failures. Kept the previous prompt instead of shipping vibes over evidence. + +### Verified + +- `showrunner`: `tests/test_authored_shots.py tests/test_llm_scene_planner.py` -> `43 passed`. +- `showrunner`: full suite -> `233 passed`. +- `showrunner`: fallback/autonomous/planner focused suite -> `44 passed`. +- `showrunner`: autonomous-loop/layout focused suite -> `25 passed`. +- `showrunner`: planner/director/executor/scout focused suite -> `70 passed`. +- Real LLM planner eval, no HeyGen, GitHub repo scenario, 2 turns, screenshot + viewport + cue execution: + - `ok_turns=2`, `failed_turns=0`, + - author latency mean `8.36s` (`5.91s` min, `10.80s` max), + - LLM-call latency mean `5.57s`, + - estimated speech `18.0s` and `19.2s`, + - host voice heuristic pass rate `1.0`. +- Real LLM planner eval, no HeyGen, five scenarios, 2 turns each, screenshot + page text + visible elements + scroll state + cue execution: + - `/tmp/showrunner_prompt_iter4_20260525/20260525T215729Z`, + - `ok_turns=10`, `failed_turns=0`, + - author latency mean `11.02s` (`5.65s` min, `13.88s` max), + - LLM-call latency mean `4.79s`, mode bucket `6.0s`, + - estimated speech mean `18.4s` (`16.8s` min, `19.6s` max), + - cue ops: `highlight=9`, `scroll=5`, `point=3`, `navigate=2`, `click=1`, + - executed browser cues: `20/20` succeeded. +- Failed prompt experiment, same five-scenario eval: + - `/tmp/showrunner_prompt_iter5_20260525/20260525T220226Z`, + - `ok_turns=8`, `failed_turns=2`, + - failures were too-short scene repair and invalid post-navigation click ordering, + - experiment reverted. +- `git diff --check` -> passed. +- Headed layout probe: + - fixed OBS viewport: configured `1920x1080`, executor viewport `1920x1080`, screenshot `1920x1080`, + - native fullscreen probe reproduced the bad macOS viewport shape: executor viewport `1496x776`, screenshot `2992x1552`. + +## 2026-05-24 + +### Implemented + +- Increased the active runway baseline from `20s` to `30s`: + - showrunner push gate defaults, + - direct-producer `max_runway_sec` defaults, + - run scripts, + - deterministic runway proof. +- Lengthened the startup seed to 112 words, about `44.8s` at `2.5 words/sec`, so the first real Claude scene has roughly the same 15s planning margin before the 30s gate opens. +- Fixed the real continuity bottleneck: the planner may now prepare the next scene while one pushed scene is still in flight. `max_in_flight_scenes=1` still limits what has already been sent to HeyGen; it no longer prevents Claude from drafting the next near-term scene. +- Split scout navigation into two explicit modes: + - default `shadow_preflight`: scout chooses the URL, a hidden browser loads/screenshots it for Claude, and OBS-visible Chrome stays put until the avatar speaks a timed `navigate` cue, + - `timed_handoff`: same visible timing behavior, but without hidden preflight analysis, + - `immediate_observe`: the older eval/development behavior that navigates before narration so Claude can author from the destination page immediately. +- Added page-handoff preplanning safety: if an in-flight scene contains a page-changing cue, the planner waits until that cue resolves before preparing the next scene. Non-page-changing scenes can still preplan ahead for voice continuity. +- Added a real shadow-preflight smoke with saved screenshots. It proves the OBS-visible browser remains unchanged during hidden preflight, while Claude receives the hidden target page and screenshot before the timed navigate cue moves the visible stage. +- Removed legacy producer-idle/silence fields from active M1 debug/runway payloads and the debug UI. Runway is the only active hot-path push gate. +- Browser cursor polish: + - page-changing clicks now detect URL changes and settle the cursor after navigation, + - navigate/back/forward/new-tab/close-tab/switch-tab/address-bar flows now land the cursor in the central viewing rectangle, + - page-change wander is clamped inside that central rectangle instead of drifting toward dead corners. +- Debug UI now separately shows browser action count and browser error count from showrunner events, so click/navigate/highlight execution is easier to verify alongside cue marker fires. +- Fixed direct-showrunner runway truth: `/producer/runway`, `/debug/state`, and `/producer/push_segment` now prefer the wrapper/player `video.currentTime` sample when the player clock is fresh. During HLS stalls, runway stays pinned to the frozen media clock instead of draining by wall time. +- Added `player_clock_ready` / `runway_clock_source` telemetry. Showrunner pushes now wait while the player clock is missing/stale, so cue-bearing scenes are not sent before `word_tracker` can map words to the OBS-visible media clock. +- Hardened `word_tracker` cue firing to use sampled `video.currentTime` when available and reject player-clock anchors older than `2s`; stale anchors no longer fire browser commands by wall-clock extrapolation. +- Expanded the LLM planner contract so Claude knows the current cue vocabulary: wait/wander durations, new-tab URL behavior, switch-tab targets, back/forward/close-tab, keyboard ops, `google_search`, and indexed `element:N` targets. + +### Verified + +- `showrunner`: full suite -> `219 passed`. +- `stream_orchestrator`: full suite -> `135 passed`. +- `script_producer`: full suite -> `46 passed`. +- `script_producer`: `tests/test_word_tracker_timing.py` -> `23 passed`. +- `tools.probe_sync_and_runway` -> `ok: true`; `30.01s` runway refuses push, `30.0s` accepts push, and browser cue firing still waits for player-clock truth. +- Focused BrowserExecutor smoke confirmed both click-navigation and direct navigation leave the visible cursor inside the central viewing rectangle. +- `debug.html` inline JavaScript passes `node --check`. +- Shadow-preflight screenshot smoke: + - visible before and visible after preflight are byte-identical, + - hidden preflight loaded the target page, + - planner received the target page URL/title plus a `24,219` byte screenshot, + - simulated timed navigate cue then moved the visible browser to the target page. + +## 2026-05-23 + +### Implemented + +- Added operator run scripts: + - `scripts/01-login-browser-profile.sh` for manual Google login with the dedicated persistent Chrome profile, + - `scripts/02-start-avatar.sh` for M1/external producer mode, + - `scripts/03-run-autonomous-show.sh` for the real headed autonomous loop with current pacing defaults, + - `scripts/04-open-local-views.sh` for local viewer/debug pages, + - `scripts/00-stop-local-processes.sh` for stuck local process cleanup. +- Added `research/CLOUD_BROWSER_STRATEGY.md` after reviewing Bright Data Agent Web Access / Browser API and Browser Use Cloud. Decision: copy the remote-CDP/profile/proxy/live-preview strategy later, but keep local headed Chrome as the default v1 OBS-visible runtime until remote CDP proves media-clock sync and capture quality. +- Refined browser-command polish from operator feedback: + - scroll no longer moves the cursor into the center if the cursor is already inside the central reading rectangle, + - Google search no longer parks the cursor in the bottom-left after results load; it moves to a natural viewing posture on the results page, + - default cursor wander increased from `7px/2 steps` to `10px/3 steps`, + - the lab now has separate `New Blank Tab` and `New Tab URL` buttons so "new fresh tab" is not confused with opening the URL field in a new tab, + - `BrowserExecutorConfig.user_data_dir` plus `SHOWRUNNER_BROWSER_USER_DATA_DIR` / `--browser-user-data-dir` allow an opt-in dedicated Chrome profile for persisted Google sign-in. +- Updated the current dedicated-queue pacing baseline: + - `seed_text` is now 112 words, about `44.8s` at 2.5 words/sec. + - Disabled bootstrap continuation text. + - `max_silent_sec` is now `0` and is a no-op in M1 pacing. + - Removed the active showrunner producer-silence force-push path. + - Default push gate is now `30s`. + - Default max in-flight scenes is now `1`. + - Scene target is now `17-20s`, about `43-50` words at `2.5 words/sec`. + - Segment compiler hard cap is now `20s`; the LLM planner rejects below `17s` and asks once for repair. + - Claude temperature default is now `0.5`. + - Browser navigation/action Playwright timeouts default to `0`, meaning no client-side timeout. +- Updated deterministic browser-director fallback narration so non-authored fallback scenes pad into the 43-50 word band instead of producing tiny chunks. +- Disabled fake-success sync fallbacks in `WordTracker`: + - no fixed wall-clock Path B when no player clock exists, + - no tail-timeout firing for remaining cues, + - no player-clock stall/absolute fallback fire. +- If a word timestamp exists but the player anchor has not arrived yet, the cue now waits for a matching player clock. If the word timestamp itself is missing, the cue is logged as unmappable and is not fired. +- Added `showrunner/tools/eval_llm_scene_planner.py`, a real-browser LLM eval harness that mirrors the autonomous loop's scout -> navigate -> observe -> screenshot -> author path and stores sanitized raw requests, raw responses, latency, parsed scene, scores, and optional cue-execution screenshots. +- Added `showrunner/tools/browser_command_lab.py`, a headed Chrome command lab with a local control UI, built-in fixture page, motion knobs, visible target suggestions, action log, screenshots, and buttons for every current browser cue op. +- Added explicit browser-lab operator controls to relaunch headed Chrome and bring Google Chrome to the front, because the previous lab started Chrome implicitly and made the controlled browser easy to miss. +- Improved browser action realism after operator QA: + - scroll now uses the actual native Chrome viewport height, central cursor posture, smaller `small_up/small_down` variants, and eased wheel deltas instead of constant-speed chunking, + - scroll keeps the normal cursor icon instead of pretending the page is being grabbed, + - cursor actions now include configurable idle wander/micro-movement and a larger explicit `wander` action, + - highlight now performs a visible drag-style selection before applying DOM/native selection, + - typing moves to the field, clicks it, then types with per-character delay, + - page snapshot targets are deduped by visible text and include indexes, selectors, geometry, and affordances such as `type`, `click`, `navigate`, and `read`, + - lab config inputs no longer reset while the operator is editing them, + - added manual keyboard commands: Enter, Backspace, Delete, hotkey, wander, URL shortcut/go, and `google_search`. +- Fixed confusing lab command routing: + - `URL Shortcut / Go` now uses only the URL field and rejects DOM targets such as `text=Jump to deep section`, + - `New Tab URL` now uses URL/Value instead of the selected DOM target, + - `Switch Tab` now uses the Value field or clickable tab pills, and the executor calls `bring_to_front()` so the visible Chrome tab actually changes. +- Added first-class `google_search` browser cue. It opens Google in a new tab, types the query into the search box when possible, and lands on the result page. Planner instructions treat it as a final handoff; the next scene must observe the results before narrating them. +- Reviewed Browser Use and OxyMouse implementation patterns: + - copied the Browser Use idea of indexed, affordance-rich browser state for LLM clarity, + - copied Browser Use's scroll-context idea by adding page-depth telemetry to snapshots and the LLM payload, + - kept our synced cue executor instead of adopting Browser Use as the runtime because Browser Use does not know about avatar media-clock cue timing, + - copied OxyMouse-style ideas: Bezier paths, acceleration/deceleration phases, corrective micro-movements, and random movement testing; did not add the dependency yet. +- Downloaded durable local references outside the repo: + - `/Users/ularkimsanov/Desktop/browser-agent-references/browser-use` + - `/Users/ularkimsanov/Desktop/browser-agent-references/OxyMouse` +- Added `showrunner/showrunner/movement_model.py`, a dependency-free movement layer inspired by OxyMouse phases: + - `cursor_path()` for accelerated/cruising/decelerating/corrective cursor travel, + - `wheel_deltas()` for eased scroll wheel events, + - `wander_targets()` for visible browsing drift. +- Added `research/BROWSER_AGENT_PATTERNS.md` with the local Browser Use/OxyMouse review and decisions. +- Hardened the browser overlay so cursor/scroll/highlight actions re-inject the overlay after page loads instead of failing when `window.__showrunnerOverlay` disappears. +- Added first-class `element:N` / `index:N` / `#N` viewport-element targets: + - `snapshot_page()` now includes `scroll_state` with pages above/below and vertical page percentage, + - the planner prompt now prefers `element:N` for visible point/highlight/click/type cues and receives `scroll_state`, + - the browser director resolves LLM-authored indexed targets before safety validation, + - the browser executor resolves indexed targets at runtime for manual lab/API cues, + - the command lab target suggestions now set `element:N` and show page-depth telemetry. +- Added `showrunner/tools/probe_sync_and_runway.py`, a deterministic developer proof for the two hard gates: runway-only pushing and player-clock-only cue firing. +- Tightened planner prompt control after real evals: + - review modes must include one concrete improvement beat, + - authored shots now have explicit per-shot word ceilings, + - length-repair prompts now target exactly 2 balanced shots and `48-50` spoken words. +- Bounded `LLMScenePlanner` raw message history to the latest `8` messages by default. Long-term continuity should come through compact `PromptMemory`, not unbounded raw JSON replay. +- Fixed authored page-handoff validation to compare page-changing cues against the actual observed browser snapshot URL instead of the model's top-level requested URL. +- Added one-shot planner repair for page-changing handoff errors and malformed/targetless authored cues, not only overlong/short segment length. + +### Verified + +- `showrunner`: `tests/test_segment_plan.py tests/test_runtime.py tests/test_llm_scene_planner.py tests/test_autonomous_loop.py tests/test_browser_director.py` -> `71 passed` +- `stream_orchestrator`: `tests/test_pacing.py tests/test_feeder.py tests/test_config.py tests/test_direct_producer.py` -> `42 passed` +- `script_producer`: `tests/test_word_tracker_timing.py` -> `22 passed` +- `showrunner`: `tests/test_llm_scene_planner.py` -> `18 passed` +- `showrunner`: `tests/test_llm_scene_planner.py tests/test_authored_shots.py tests/test_browser_director.py` -> `51 passed` +- `showrunner`: `tools.probe_sync_and_runway` -> `ok: true` +- Browser command lab fixture API smoke: + - point, click, type, highlight, scroll up/down, anchor click, back, forward, new tab, switch tab, close tab, wait, screenshot all returned successful action records. + - post-QA smoke: config edit, small smooth scroll with default cursor, drag-style highlight, click-before-type, larger wander, invalid URL rejection, new tab, visible switch tab, and Google search all returned expected action records. + - movement-model smoke: small scroll, wander, and click all returned successful action records after restarting the headed lab. +- `showrunner`: `tests/test_movement_model.py tests/test_bezier_path.py tests/test_browser_executor.py tests/test_browser_scout.py tests/test_curiosity_ops.py tests/test_authored_shots.py tests/test_llm_scene_planner.py` -> `76 passed` +- `showrunner`: `tests/test_browser_targets.py tests/test_browser_scout.py tests/test_authored_shots.py tests/test_llm_scene_planner.py tests/test_browser_executor.py` -> `62 passed` +- `showrunner`: `tests/test_movement_model.py tests/test_bezier_path.py tests/test_browser_executor.py tests/test_browser_scout.py tests/test_curiosity_ops.py tests/test_cues.py tests/test_browser_pipeline.py tests/test_authored_shots.py tests/test_browser_director.py tests/test_llm_scene_planner.py` -> `104 passed` +- Browser command lab index/scroll smoke: + - `point element:4` resolved to the fixture button, + - `type element:1` resolved to the fixture input, + - `scroll down` moved the viewport from top-of-page to about `62.2%` depth and reported updated `pages_above/pages_below`. +- Updated stale buffer/scroll tests to match the current runway-only and smoother-scroll behavior. +- Tightened page-changing handoff enforcement: + - clicks on visible elements with `navigate` affordance or `href` are now treated as page/session handoffs, + - link-like clicks must be the final authored shot unless a later planner pass has observed the new page, + - this prevents the avatar from clicking into a new file/page and immediately narrating unobserved content. +- Real LLM Browser Use-style contract eval, no HeyGen: + - `/tmp/showrunner_llm_evals_index_contract/20260524T054613Z`: `3/3` valid; author turn mean `9425.99ms`, all Claude-call mean `4020.57ms`, vision author-call mean `4711.61ms`. + - `/tmp/showrunner_llm_evals_handoff_fix/20260524T055240Z`: GitHub Trending rerun valid `17.6s`; final cue is the `CLAUDE.md` click handoff, with no post-click unobserved scroll. +- Autonomous-loop smoke, no HeyGen: + - `tools.probe_sync_and_runway` -> `ok: true`, with no push above `30.0s`, push at `30.0s`, and no browser cue fire before player clock. + - `tools.smoke_autonomous_loop_fake` -> 3 browser actions / 3 marker events, runtime chose `draft_more`, `commit_from_draft`, `push_to_heygen`. + - `tools.smoke_autonomous_loop_multi_fake` -> `ok: true`, 3 scenes completed, 7 browser actions, 7 marker events. +- Real HeyGen one-scene run: + - Session `fd3585e90b4a4acb89ebc064991516a4` warmed to streaming in `12.34s`. + - Wrapper player clock was healthy and cues fired through `A-player`. + - Marker alignment drift: about `45.8ms`, `42.3ms`, and `39.6ms`. + - Planner fell back to the old demo fake scene after primary overlong/repair failure; that pushed invalid demo targets onto GitHub Trending. + - Cleanup posted `final=true`; no terminal status inside 5s; `/cancel` returned `cancelled=true`. +- Replaced real-run deterministic fallback with `ViewportSafeFallbackSceneClient`: + - no page-specific fake selectors, + - only `wait` plus `scroll` on the current page, + - generic narration that buys time without inventing receipts. +- `showrunner`: full suite -> `211 passed` +- Browser-command polish verification: + - focused executor/movement tests -> `20 passed`, + - persistent-profile startup smoke passed with a temporary profile, + - command-lab API smoke confirmed `New Blank Tab` opens `about:blank`, + - command-lab API smoke confirmed Google search cursor lands near the results area instead of bottom-left, + - command-lab API smoke confirmed scroll does not move the cursor when it starts inside the central reading rectangle. +- `showrunner`: full suite -> `212 passed` +- Real LLM evals, no HeyGen: + - scout path `/tmp/showrunner_llm_evals_real/20260524T021015Z`: valid `19.2s` / `48` words, scroll + highlight executed successfully, authoring path `10008.48ms`. + - no-scout path `/tmp/showrunner_llm_evals_real/20260524T015809Z`: valid `20.0s` / `52` words, scroll + highlight executed successfully, authoring path `10351.4ms`. + - four-scenario batch `/tmp/showrunner_llm_evals_real_batch/20260524T032348Z`: `3/4` valid before the handoff/repair fixes; all Claude calls mean `4315.96ms`, mode bucket `2000ms`; vision author calls mean `5618.01ms`, mode bucket `4000ms`. + - post-fix Wikipedia rerun `/tmp/showrunner_llm_evals_real_batch/20260524T033002Z`: valid `18.8s`; all Claude calls mean `4267.17ms`, mode bucket `2000ms`; vision author calls mean `5188.6ms`, mode bucket `4000ms`. + +### Still Pending + +- Run one bounded real HeyGen E2E with the new ~45s seed / 30s runway gate and confirm cleanup reaches terminal status or `/cancel`. +- Ask HeyGen whether a single realtime HLS session's `video.currentTime` is expected to be monotonic and how `/words` timestamps map across playlist discontinuities/currentTime resets. +- Add a short media-only validation path with wrapper watchdog disabled/enabled before another full LLM+HeyGen burn. +- Add startup preplanning before stream create if the ~45s seed is still not enough. +- Cue schema unification between `showrunner` and `script_producer`. +- Visual QA screenshot assertions for real headed Chrome cursor/selection/click/scroll polish. +- Real LLM tone pass under the new 17-20s scene band. + +### Historical Notes From Earlier 2026-05-23 Work + +- Tuned the responsive pacing baseline: + - Shortened the startup seed/continuation text so the first real scene does not sit behind a long canned monologue. + - Lowered default autonomous push gate to `14s`, polling to `0.5s`, scene target to 8-12s, and hard segment cap to `16s`. + - Added `--heygen-push-below-sec`, `--push-poll-interval-sec`, and `--max-in-flight-scenes` to `tools.run_autonomous_loop`. + - Added planner timing logs so real runs show how long Claude/browser planning took. +- Added one-shot overlong-scene repair in `LLMScenePlanner`. + - If Claude returns an otherwise valid scene that exceeds the hard speech cap, the planner asks once for corrected JSON at 24-30 words. + - Failed overlong attempts are still not appended to conversation history. +- Added primary-player clock hygiene: + - `wrapper.html` now generates a `client_id` for playback anchors and supports `?clock=off`. + - `SessionClock` elects the first fresh player as primary and ignores side-player anchors until the primary goes stale. + - `tools.measure_media_playback` defaults to `--clock-mode off` and now writes a summary on SIGINT/SIGTERM. +- Reviewed the copied HeyGen PR #37607 reference update under `api_references/`. + - Confirmed the idle-timeout change is server-side `avatar_realtime_runtime_config`, not a new create-session request field. + - Confirmed the normal clean finish path remains `/text` with `final=true`. + - Confirmed the new hard-abort path is `POST /v3/avatar-realtime/{stream_id}/cancel`. +- Added M1 support for the new `/cancel` endpoint. + - `ApiClient.cancel_session()` calls the endpoint and returns the `cancelled` boolean. + - `Session.cancel()` exposes the hard-abort path. + - Supervisor teardown still prefers `final=true`, then falls back to `/cancel` if final cannot be posted. +- Added `stream_orchestrator/tools/probe_idle_timeout.py`. + - Creates a real text-stream session, waits without extra text, and reports whether the session survives past a threshold. + - This is the one-command proof to run after HeyGen applies the long `idle_timeout_sec` runtime config. +- Added the default two-phase autonomous planning path: + - scout a safe public URL/topic first, + - navigate the real browser, + - observe the actual page, + - then author narration/cues from visible state. +- Added `--one-shot-planner` as an escape hatch; default runs two-phase. +- Added restrained host-personality guidance to the planner: dry, observant, useful, bounded curiosity. +- Added `HOST_CHARACTER.md` as the durable show bible for the avatar's character, roast rules, show formats, ratings, and guardrails. +- Upgraded the planner's runtime personality from generic "dry host" to Live Critic Desk: + - evidence-first software satire, + - Repo Roast Court / Landing Page Smell Test / Docs Interrogation / Demo Theater Watch / Repair Minute modes, + - artifact-focused roasts with one useful repair note, + - optional ratings such as `README Wearing A Blazer` and `Grandma With Claude Code Could Ship This`. +- Changed the default autonomous-loop show goal from public-web research to public-web software critique. +- Tightened authored-shot prompt rules after a real LLM smoke emitted a targetless `highlight` cue; the planner now explicitly says targetless authored cues are rejected. +- Added `tools.smoke_browser_scene_matrix` to run real LLM browser-scene QA across multiple public websites and save viewport screenshots after each cue. +- Tightened browser-scene reliability from the matrix findings: + - visible text targets now prefer a visible match instead of the first offscreen match, + - `highlight` can auto-scroll the target into view when the planner slightly undershoots, + - planner scenes are compiled/length-validated before history is appended, + - authored anchors fall back to the first words of the shot when the model supplies a bad anchor, + - positive authored `lead_ms` values are normalized to safe non-positive defaults, + - page-changing cues are now treated as handoffs to the next observed scene. +- Enforced explicit page-changing authored cues as final-shot handoffs in the director, so the next planner turn must observe the new page before narrating details from it. +- Shortened LLM scene target length to 10-16 seconds / under 42 words and lowered the hard segment cap to 22 seconds so the stream reacts faster and avoids stale buffers. +- Added immediate speech-tracking failure recovery: + - `WordTracker` now keeps pushed segments as a FIFO, + - reports active and pending segments abandoned on session swap, + - the autonomous loop turns those reports into immediate scene failures instead of waiting for the full browser-action timeout. +- Added prompt/source-choice refinements from the transcript pass: + - avoid random catchphrases, + - use setup -> receipt -> joke -> repair note, + - avoid Hugging Face model pages for v1 unless already observed as public/readable, + - keep quotes short and visible. +- Added a selector reliability guard for authored highlight/point cues: + - empty targets are rejected, + - long unstructured highlight targets are rejected, + - planner prompt now prefers exact selectors or short exact visible-element text. +- Added safe voice-control tag handling: + - `spoken_text` keeps tags such as `[sighs]` for the voice engine, + - `word_tracking_text` strips them for cue alignment, + - `WordTracker` ignores bracket/SSML control tokens if they appear in `/words`. +- Kept captions aligned with that same policy: captions filter the broader voice-control token set, and the caption typography now uses zero letter spacing. +- Added operator debug visibility: + - M1 now accepts `POST /debug/showrunner_event` and includes recent showrunner debug events in `/debug/state`. + - `tools.run_autonomous_loop` mirrors sanitized Claude requests, raw Claude responses, loop config, and browser executor outcomes into the M1 debug page. + - `BrowserPipeline` has an optional debug callback for action/error outcomes. + - `/debug` now displays last Claude request, raw response, latency, last pushed HeyGen text/markers, cue-fire stream, word stream, media drift, and last browser executor outcome. +- Updated root `PLAN.md` with the current working/not-working state, exact runtime constants, media-continuity blocker, and current debug UI contract. + +### Verified + +- Responsive pacing patch checks: + - `showrunner/tests/test_llm_scene_planner.py tests/test_autonomous_loop.py tests/test_media_telemetry.py` -> `39 passed` + - `stream_orchestrator/tests/test_session_clock.py tests/test_direct_producer.py tests/test_captions_page.py` -> `39 passed` + - `tools.measure_media_playback --help` verified the new `--clock-mode` flag. + - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - Real Anthropic/browser smoke without HeyGen passed on GitHub Trending; it produced a short two-shot scene with a final page-changing handoff. +- Latest dedicated-queue real E2E sample: + - First request warmup was `43.16s`. + - First real content push had runway `-1.09s -> 14.37s`. + - Loop stopped with `planner_failure_alarm` because Claude returned several 17-25s scenes against the new 16s hard cap. + - Media summary showed median lag `5.38s`, mean `7.30s`, one HLS currentTime reset, five stalls, max stall `11.65s`. + - Marker drift for fired markers remained good: mean `8.78ms`, max `17.57ms`. + - The old stream `b7456496b41244058614ed86dc0459e0` is confirmed `completed` with `end_reason=final_marker`; no cancel was needed. +- Second dedicated-queue validation run: + - Started showrunner earlier and reduced in-flight scenes to 1 to test startup pre-planning. + - Warmup was `9.91s`. + - First content push still arrived late: runway `-7.85s -> 7.56s`. + - Media summary was bad: `continuous=false`, mean lag `42.04s`, median lag `41.68s`, max lag `76.34s`. + - HLS had 4 backward jumps to `0`, 13 stalls, `79.61s` total stall time, max stall `12.92s`, observed playback rate `0.452x`. + - Marker paths were mixed: 1 normal `A-player`, 1 `A-player-fallback`, 1 `tail-timeout`. + - Conclusion: stop tuning lower than 14s until HLS continuity / `/words` timeline mapping is clarified. +- Debug visibility patch checks: + - `stream_orchestrator/tests/test_debug_page.py tests/test_direct_producer.py` -> `20 passed` + - `showrunner/tests/test_browser_pipeline.py tests/test_llm_scene_planner.py tests/test_autonomous_loop.py` -> `43 passed` + - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`, proving the debug bridge fails silent when no M1 debug receiver is available. +- Focused post-audio-tag tests: + - `showrunner/tests/test_segment_plan.py tests/test_producer_client.py` -> `14 passed` + - `script_producer/tests/test_word_tracker_timing.py` -> `12 passed` +- Latest full suite baseline from this patch series: + - `showrunner`: `178 passed` + - `stream_orchestrator`: `127 passed` + - `script_producer`: `38 passed` +- Post-PR #37607 integration checks: + - `stream_orchestrator/tests/test_client.py tests/test_supervisor_recovery.py` -> `6 passed` + - `uv run python -m tools.probe_idle_timeout --help` verified. + - `stream_orchestrator` full suite -> `127 passed` + - `showrunner` full suite -> `178 passed` + - `script_producer` full suite -> `38 passed` +- Latest local browser smokes from this patch series: + - `tools.smoke_browser_executor` passed. + - `tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. +- Real Anthropic planner smokes: + - `tools.smoke_llm_scene_planner --real` passed. + - `tools.smoke_llm_scene_planner --real --scout-first` passed and returned a safe Playwright docs scout URL. + - `tools.smoke_llm_scene_planner --real --scout-first --browser-observe --headed` passed: scouted Playwright docs, opened headed Chrome, observed the real page, and authored accepted cues from that snapshot. + - Re-ran `tools.smoke_llm_scene_planner --real --scout-first --browser-observe --headed` after the Live Critic Desk prompt update. It chose `https://github.com/microsoft/playwright`, opened the real repo, produced a Repo Roast Court segment, praised the README when the evidence was good, and authored valid scroll/highlight cues for visible README text. + - Re-ran after the personality transcript pass. It again chose `https://github.com/microsoft/playwright`, authored valid scroll/highlight cues, and produced a specific evidence-first segment at `18.97s` / `50 words`. Tone is usable; length still needs tightening toward 35-42 words. +- Real headed Chrome browser-scene matrix: + - `tools.smoke_browser_scene_matrix --real-llm --headed` passed across `github_trending`, `github_repo`, `github_click_file`, `docs_quickstart`, `wikipedia`, and `arxiv`. + - Covered navigate, scroll, highlight, auto-scroll-to-highlight, and a real visible click on `CLAUDE.md`. + - Cursor overlay was visible for every recorded action. + - Viewport screenshots are in `/tmp/showrunner_scene_matrix`. +- Real HeyGen canary smoke: + - Started `stream_orchestrator` against `api-canary.heygen.com`. + - Ran `tools.smoke_real_loop --timeout-sec 180`. + - Result: `typed="real canary loop"`, `browser_actions=3`, `marker_events=3`, pushed after `67.42s`. + - Marker fire path was `A-player` for all 3 cues with drift about `0.3ms`, `9.0ms`, and `33.1ms`. + - Inspected `/tmp/showrunner_real_loop.png`; typed text and native text-selection highlight were visible. + - Stopped the orchestrator after the smoke; port 8765 was confirmed closed. +- Real headed autonomous HeyGen run after FIFO/session-ready patches: + - Command: `tools.run_autonomous_loop --real-llm --headed --layout-preset obs-1080p --autonomy-mode curious --max-scenes 3 --timeout-sec 480 --producer-silence-force-push-sec 20` + - Result: `ok=false`, `stop_reason=browser_failure_alarm`, `scenes_pushed=3`, `scenes_completed=0`, `marker_events_total=0`, `browser_actions_total=0`. + - Browser navigation itself worked; final screenshot was `/tmp/showrunner_e2e_final.png`. + - The current canary environment still rotated sessions from `idle_timeout`, causing `WordTracker` to abandon active/pending segments before markers fired. + - Conclusion at the time: the long-running demo was blocked on server-side queue/idle config, not on headed Chrome. +- HeyGen PR #37607 idle probe: + - `gh pr view 37607 --repo heygen-com/experiment-framework` reports the PR is merged. + - Ran `STREAM_ORCHESTRATOR_EXTERNAL_PRODUCER_MODE=1 uv run python -m stream_orchestrator` against canary with no showrunner pushes. + - Canary still returned `completed/end_reason=idle_timeout` after about `36.19s` and again after about `38.37s`. + - Clean Ctrl-C on the third session posted `final=true` successfully with `byte_drift=0`; `/cancel` fallback was not needed in that shutdown. + - Conclusion: the code is merged, but this account/space still needs the server-side `avatar_realtime_runtime_config` change, likely `idle_timeout_sec: 86400` or another high positive value. +- HeyGen long-idle confirmation after the dev applied the per-space config: + - `uv run python -m tools.probe_idle_timeout --min-survival-sec 90` + - Result: `ok=true`, `reason=survived_minimum_without_idle_timeout`, `status=streaming`, `end_reason=null`. + - The session survived the old 36-38s failure window and the 90s probe window; cleanup posted `final=true`. + - Production showrunner runs should use runway-only pacing with `--producer-silence-force-push-sec 0`. +- Headed Chrome viewport fix: + - Confirmed the old headed launch reported `innerWidth=1920`, `innerHeight=1080` while the real Chrome window was only about `1476x843`, so Playwright screenshots were not trustworthy for OBS fit. + - Changed headed `BrowserExecutor` to use Chrome's native visible viewport by default. + - Changed `BrowserExecutor.screenshot()` to viewport-only so visual QA shows what viewers see, not the whole page. + - Added `--emulated-viewport` as an explicit escape hatch for fixed Playwright viewport geometry. + - Verified patched headed Chrome reports `innerWidth=1496`, `innerHeight=776`, and `BrowserExecutor.viewport_size()` returns `1496x776` on this Mac display. + - Focused tests: `showrunner/tests/test_browser_executor.py tests/test_director_polish.py` -> `27 passed`. +- Added stale avatar-player clock filtering: + - An old wrapper tab was still posting `/control/video_started` anchors for a previous HeyGen session id after orchestrator restart. + - M1 now ignores player-clock anchors unless the posted `session_id` matches the current active session. + - Focused tests: `stream_orchestrator/tests/test_direct_producer.py tests/test_client.py tests/test_supervisor_recovery.py` -> `16 passed`. +- Updated HeyGen runtime assumption from "unlimited idle" to the dev-confirmed dedicated queue contract: + - Dedicated queue is enabled for this space. + - No-audio kill window is 30 minutes. + - There is only one host, so every restart must release the current stream first. + - If dedicated queue produces corrupted video, ask HeyGen to flip `use_dedicated_queue=false` for this space. +- Tightened teardown for the dedicated-host reality: + - M1 now posts `final=true`, waits briefly for terminal status, and calls `/cancel` if the session still reports `streaming`. + - This covers the observed behavior where `final=true` was accepted but status remained `streaming` for several seconds. + - Focused tests: `stream_orchestrator/tests/test_supervisor_recovery.py tests/test_direct_producer.py tests/test_client.py` -> `17 passed`. + +### Historical Pending Before Latest Pacing Patch + +Superseded by the current `Still Pending` list at the top of this date section. + +## 2026-05-22 + +### Added + +- Created this root `CHANGELOG.md` as the durable handoff log for Codex and Claude Code. +- Created root `PLAN.md` with the current architecture, findings, sync contract, demo direction, and next patch list. + +### Audited + +- Reviewed current repo modules while avoiding `ai-twitch-streamer`. +- Reviewed: + - `stream_orchestrator` + - `showrunner` + - `script_producer` + - `api_references` + - root docs: `README.md`, `STATUS.md`, `HANDOFF.md`, `TECHNICAL_OVERVIEW.md` +- Reviewed the local Poke files supplied from Downloads and summarized usable personality principles without copying prompt text into the app plan. +- Reviewed current browser-control ecosystem sources: Browser Use, Browserbase, Agent Browser, BrowserMCP, Playwright, Selenium Actions, OxyMouse, HumanCursor, DeepMind AI Pointer, VedalAI Neuro SDK, and Excalidraw MCP. + +### Verified + +- Unit tests: + - `stream_orchestrator`: `uv run pytest -q` -> `123 passed in 68.81s` + - `showrunner`: `uv run pytest -q` -> `157 passed in 1.28s` + - `script_producer`: `uv run pytest -q` -> `34 passed in 0.34s` +- Fake/local smokes: + - `showrunner/tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - `showrunner/tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - `showrunner/tools.smoke_browser_executor` passed all local actions: navigate, highlight, point, type, click, scroll, highlight. + +### Findings + +- P0: The stream can still hit HeyGen `idle_timeout` in external producer mode. + - `showrunner/showrunner/runtime.py` has a force-push path based on producer silence. + - `showrunner/showrunner/autonomous_loop.py` waits only for low runway before calling that path. + - `stream_orchestrator/stream_orchestrator/supervisor.py` exits on external-producer `idle_timeout`. + - Real log evidence: the latest run accepted pushes at `22:10:44Z`, `22:11:07Z`, and `22:11:44Z`, then ended with `idle_timeout` at `22:12:05Z`. +- P1: Browser sync itself is not the main blocker. + - Marker logs show `A-player` cue firing with drift roughly `10ms`, `23ms`, `44ms`, and `48ms`. + - `script_producer/script_producer/word_tracker.py` already follows the latest player clock anchor instead of relying on stale fixed wall-clock sleeps. +- P1: The autonomous LLM planner is too one-shot. + - It can choose a URL and also author page-specific cues before the browser has inspected that URL. + - This should become a two-phase scout/observe/author/validate loop. +- P2: Cue schemas are duplicated and divergent. + - `showrunner/showrunner/cues.py` supports tab/history ops. + - `script_producer/script_producer/cues.py` only supports the original v1 browser ops. +- P3: `BrowserPipeline` waits again for positive `lead_ms`, even though `WordTracker` already applies lead timing. +- P3: Root docs are partially stale. Some documented open issues are fixed in code, and some paths refer to older layouts. + +### Current Recommendation + +- Keep the v1 demo as a constrained local Playwright "live research desk" with synced highlights, scrolls, and public-page navigation. +- Do not switch the core runtime to a generic browser-agent framework yet. +- Use browser-agent tools only as research/scout inspiration until streaming stability is solved. + +### Next Patch + +- Fix producer-silence wakeup in `showrunner/showrunner/autonomous_loop.py`. +- Make external-producer `idle_timeout` recoverable in `stream_orchestrator/stream_orchestrator/supervisor.py`. +- Add a bridge segment fallback for silence danger when no committed segment is ready. +- Add tests for high runway plus high producer silence. +- Then run a real 10-30 minute stream soak before changing the personality or demo concept further. + +### Implemented + +- Completed first Phase 0 stream-stability patch: + - `AutonomousLoopConfig` now has `producer_silence_force_push_sec`. + - `_wait_until_runway_low()` can wake on producer-silence danger when that fallback is enabled. + - Default showrunner behavior is now runway-only for long-idle/dedicated-queue sessions. + - External-producer `idle_timeout` now logs recovery intent instead of ending the supervisor loop. + - Added regression tests for high-runway/high-silence pusher wakeup and external idle recovery. +- Added a multi-phase roadmap to `PLAN.md` covering stream stability, browser reliability, personality, chat input, high-wow activity screens, and 24/7 operations. +- Clarified and hardened the existing real-browser direction: + - The codebase already had a real public-page path via `tools.run_autonomous_loop` and `BrowserExecutor`. + - Production activity screen should remain real headed Google Chrome on public pages. + - The dashboard/demo surface is smoke-test-only. + - `tools.run_autonomous_loop --fullscreen` now launches headed mode and resolves `--browser-channel auto` to Google Chrome. + - Added browser polish knobs for cursor speed and smooth-scroll cadence. + - Highlights now try native browser text selection before falling back to the overlay focus box. + - Cursor styling now maps common CSS cursor states to the visible overlay cursor. +- Added stream layout and autonomy controls: + - `tools.run_autonomous_loop` now has `--layout-preset` with `obs-1080p`, `obs-720p`, `laptop-16x9`, and `custom`. + - Default live layout is now `obs-1080p` (`1920x1080`) instead of the laptop/debug `1280x800` shape. + - Added `--autonomy-mode focused|curious|free` and `--show-goal`. + - Planner instructions now explicitly enforce viewport discipline: if viewers cannot see it, scroll before discussing/highlighting it. + +### Verified After Patch + +- Focused tests: + - `showrunner/tests/test_autonomous_loop.py` -> `13 passed` + - `stream_orchestrator/tests/test_supervisor_recovery.py` -> `3 passed` + - `showrunner/tests/test_runtime.py tests/test_autonomous_loop.py tests/test_browser_executor.py tests/test_bezier_path.py` -> `39 passed` + - `showrunner/tests/test_llm_scene_planner.py tests/test_vision_planner.py tests/test_runtime.py tests/test_autonomous_loop.py tests/test_browser_executor.py tests/test_bezier_path.py` -> `56 passed` + - `tools.run_autonomous_loop --help` verified the new layout/autonomy flags render correctly. +- Full tests: + - `showrunner`: `162 passed` + - `stream_orchestrator`: `123 passed` + - `script_producer`: `34 passed` +- Local smokes: + - `tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. + - `tools.smoke_browser_executor` passed navigate, highlight, point, type, click, scroll, highlight. + - Re-ran all three after the layout/autonomy/browser-polish changes; all still passed. + +## 2026-05-24 — Real Showrunner E2E Trace + +Implemented `showrunner/tools/trace_live_e2e.py`, a black-box recorder for real M1 + showrunner runs. It watches `/debug/state`, `/producer/words`, `/producer/session_clock`, `/activity/segments`, and `/control/sessions`, then writes: + +- `timeline.jsonl` +- `summary.json` +- cue/browser lag distributions +- player-clock flat runs and backward-reset detection +- LLM latency by phase +- pushed-segment runway snapshots + +Real run: + +```bash +cd /Users/ularkimsanov/Desktop/live-streamer/showrunner +uv run python -m tools.trace_live_e2e --duration-sec 420 --out-dir /tmp/showrunner_e2e_trace_20260524T103858Z --include-word-text +uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --max-scenes 5 --timeout-sec 420 \ + --event-log /tmp/showrunner_e2e_trace_20260524T103858Z/showrunner.sqlite \ + --marker-log /tmp/showrunner_e2e_trace_20260524T103858Z/markers.jsonl \ + --screenshot /tmp/showrunner_e2e_trace_20260524T103858Z/final_browser.png +``` + +Observed: + +- `5` scenes pushed and completed. +- `8` CUE events fired, all on `A-player`. +- Browser executed `7` actions and recorded `1` browser target error. +- Marker drift: mean `274ms`, min `143ms`, max `478ms`. +- Claude latency: scout mean `2.96s`; scene mean `4.84s`, max `8.55s`. +- Direct runway used `player_clock` for every push. Example proof: final old producer estimator was about `-53s` while player-clock runway was `21.73s`. +- Player clock stayed fresh, median anchor age `122ms`. +- Media/player flat runs were measurable; trace recorded `9` flat runs over `1s`, mean `9.61s`, max `20.55s`. +- A post-show teardown reset moved `video.currentTime` from about `123.79s` to `4.13s`. This happened after showrunner stopped, but it exposed a real guardrail need. + +Hardening added after the trace: + +- `SessionClock` now rejects large backward `video.currentTime` jumps from the current primary player instead of accepting a reset as fresh runway. +- Noisy per-anchor logs moved from `INFO` to `DEBUG`. +- Trace tool now handles `SIGINT`/`SIGTERM`, summarizes errors compactly, splits LLM latency by phase, reports player-clock anomalies, and auto-stops after sustained M1 `/debug/state` unreachable time. + +Follow-up issues found: + +- Planner still emitted one overlong segment: `20.80s` versus the `20s` hard cap. +- Browser cue target resolution failed once on an invisible GitHub text target: `text=clangd-lsp`. +- Fullscreen headed Chrome used the native Mac page viewport (`1700x1342` screenshot), not a fixed `1920x1080` OBS viewport. Need a layout pass. + +Verification: + +- `cd stream_orchestrator && uv run pytest tests/test_session_clock.py tests/test_direct_producer.py` -> `28 passed` +- `cd script_producer && uv run pytest tests/test_word_tracker_timing.py` -> `23 passed` +- `cd showrunner && uv run python -m py_compile tools/trace_live_e2e.py` -> passed +- Full suites after hardening: + - `stream_orchestrator`: `136 passed` + - `script_producer`: `46 passed` + - `showrunner`: `219 passed` + +## 2026-05-24 — Fixed OBS Browser Layout + +Fixed the headed Chrome layout mismatch found in the real E2E trace. + +Problem: + +- `tools.run_autonomous_loop --headed --fullscreen` previously used Chrome's native macOS viewport. +- The final E2E screenshot was `1700x1342`, not `1920x1080`. +- On Retina/native fullscreen probes, Chrome can report a `1496x776` CSS viewport and capture `2992x1552` device pixels. That is wrong for OBS and wrong for LLM screenshot consistency. + +Changes: + +- `tools.run_autonomous_loop` now uses a fixed OBS viewport by default, even in headed Chrome. +- `--native-viewport` was added as an explicit manual-debug escape hatch. +- `--emulated-viewport` remains accepted as a deprecated no-op alias because fixed viewport is now the default. +- Final run JSON now includes `browser.actual_viewport` and `browser.screenshot_size`. +- Added `tools.probe_browser_layout` to measure browser metrics and screenshot dimensions without starting HeyGen. +- Other headed browser tools now pass `use_native_viewport=False` by default: command lab, autonomous scene, scripted show, LLM eval, browser pipeline, scene matrix, and LLM smoke observe path. + +Measured probes: + +- Fixed headed OBS probe: + - command: `cd showrunner && uv run python -m tools.probe_browser_layout --headed --layout-preset obs-1080p --screenshot /tmp/showrunner_layout_headed_fixed.png` + - result: configured `1920x1080`, executor viewport `1920x1080`, screenshot `1920x1080`. +- Native fullscreen probe: + - command: `cd showrunner && uv run python -m tools.probe_browser_layout --headed --native-viewport --fullscreen --layout-preset obs-1080p --screenshot /tmp/showrunner_layout_headed_native_fullscreen.png` + - result: executor viewport `1496x776`, screenshot `2992x1552`. + +Verification: + +- `cd showrunner && uv run pytest tests/test_run_autonomous_loop_layout.py tests/test_browser_executor.py -q` -> `20 passed` +- `cd showrunner && uv run pytest -q` -> `222 passed` +- `cd showrunner && uv run python -m py_compile ...` for changed browser tools -> passed +- `cd showrunner && uv run python -m tools.run_autonomous_loop --help | rg 'native-viewport|emulated-viewport|layout-preset'` -> verified +- `git diff --check` -> passed diff --git a/README.md b/README.md index 87ecada..43dd956 100644 --- a/README.md +++ b/README.md @@ -1,151 +1,137 @@ # live-streamer -**A 24/7 AI live-streamer.** A HeyGen photo avatar hosts a continuous Twitch show: -it drives a real, visible Chrome window — navigating, scrolling, highlighting, and -clicking through live web pages — and narrates what it sees in real time. Viewers -drop GitHub repositories in chat; the avatar pulls them up and roasts them on air. - -The load-bearing trick is **word-level timestamp sync**. HeyGen's Avatar Realtime -API returns per-word timestamps for the avatar's speech, so an on-screen cursor -moves at the exact moment the avatar says "let me click here" — not five seconds -before or after. Speech, video, and browser action stay locked to one clock. - -**A representative week in production:** ~155 hours live · 417 streaming sessions · -5,765 text pushes to the avatar · ~70 ms median API-ack latency. - -## How it works - -Four independent Python packages cooperate. The showrunner is the brain; the -stream orchestrator (M1) owns the HeyGen session and the OBS-facing surfaces. - -```mermaid -flowchart TD - viewer([Twitch viewers]) -->|drop GitHub URLs| chat[Twitch chat] - chat --> ingester[chat_ingester
IRC reader + URL filter
Haiku 4.5 pre-bakes acks] - ingester -->|SQLite queue| showrunner - - subgraph showrunner["showrunner (the content brain)"] - direction LR - author[Episode author
Claude Opus 4.8] - reactive[Reactive planner
Claude Sonnet 4.6] - executor[Browser executor
Playwright] - tracker[word_tracker
cue scheduler] - author ~~~ reactive ~~~ executor ~~~ tracker - end - - showrunner -->|push scene:
narration + browser plan + cues| m1[stream_orchestrator M1
session lifecycle + HTTP/SSE] - m1 <-->|text in / audio + HLS + per-word SSE| heygen[(HeyGen
Avatar Realtime API)] - showrunner -->|navigate / scroll / highlight / click| chrome[Visible Chrome window] - - m1 -->|avatar video + captions| obs[OBS Studio] - chrome -->|window capture| obs - obs -->|RTMP| twitch([Twitch]) -``` +24/7 Twitch live-streamer: a HeyGen photo avatar narrates a live browser show. Playwright drives a real visible Chrome window through navigations, scrolls, highlights, and clicks. HeyGen Avatar Realtime v3 generates the avatar's speech with word-level timestamps so the cursor moves when the avatar says "let me click here" — not five seconds later. -## Prerequisites +This is a standalone extraction from the broader experiment-framework repo. Everything needed to run is in this directory. -- **Python 3.11** (every package pins `requires-python = ">=3.11"`) -- **[uv](https://github.com/astral-sh/uv)** for dependency / virtualenv management -- **OBS Studio** with the obs-websocket plugin (streaming + egress heartbeat) -- **Chrome / Chromium** (the showrunner installs its own via Playwright) -- A **HeyGen** API key (Avatar Realtime) and an **Anthropic** API key -- **macOS** if you want the unattended `launchd` operation described in - [docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md); the core - packages themselves are OS-agnostic +## Quick start -## Setup +Use the scripts when you do not want to remember flags. ```bash -# Fill in your keys — one root .env serves all three runtime components. -cp .env.example .env && $EDITOR .env # ANTHROPIC_API_KEY + HEYGEN_API_KEY (+ Twitch creds) +# One-time setup +cp .env.example .env && $EDITOR .env # fill ANTHROPIC_API_KEY + Twitch creds +scripts/01-login-browser-profile.sh # one-time Google login in the show's Chrome profile + # (stop with Ctrl-C before the autonomous run) + +# Terminal 1 — M1 (HeyGen session manager + OBS browser sources, port 8765) +scripts/02-start-avatar.sh + +# Terminal 2 — autonomous local headed Chrome showrunner +source .env && scripts/03-run-autonomous-show.sh + +# Terminal 3 (optional) — Twitch chat ingester (Module 4). +# Viewers drop GitHub URLs in chat, the avatar acknowledges them by name, +# navigates to the repo, and roasts it. REQUIRES `SHOWRUNNER_EPISODE_MODE=true` +# in the showrunner's env (already set if you sourced `.env`). +source .env && scripts/05-run-chat-ingester.sh -# Create each package's uv virtualenv. -cd stream_orchestrator && uv sync && cd .. -cd showrunner && uv sync && uv run playwright install chromium && cd .. -cd chat_ingester && uv sync && cd .. -cd script_producer && uv sync && cd .. +# Optional — open local viewer/debug pages. +scripts/04-open-local-views.sh -# One-time Google login in the show's dedicated Chrome profile (stop with Ctrl-C). -scripts/01-login-browser-profile.sh +# Cleanup if a local process is stuck. +scripts/00-stop-local-processes.sh ``` -The login script and the autonomous show share the same dedicated Chrome profile -(`.browser-profile` by default). Don't run them at the same time — Chrome profiles -are single-writer. Use a dedicated Google account, not your personal one. +**For unattended 24/7 operation on a Mac Mini**, see [docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md) — covers launchd plists, system settings, OBS heartbeat, and a 7-day soak checklist. -## Running the show +The login script and autonomous script share the same dedicated Chrome profile at `.browser-profile` by default. Do not run the login lab and autonomous show at the same time; Chrome profiles are single-writer state. Use a dedicated Google account, not your personal/default Chrome profile. -The `scripts/` wrappers exist so you don't have to remember flags. Bring up M1 -first; the showrunner spawns its own browser. +Manual equivalent, three terminals. M1 first; showrunner brings up its own browser. ```bash -# Terminal 1 — M1 (HeyGen session manager + OBS browser sources, port 8765) -scripts/02-start-avatar.sh -# wait for `supervisor.session_ready` in the logs +# Terminal 1 — Module 1 (HeyGen session manager + OBS browser sources) +cd stream_orchestrator +uv sync +export HEYGEN_API_KEY=... # or put it in a repo-root .env (see .env.example) +STREAM_ORCHESTRATOR_EXTERNAL_PRODUCER_MODE=1 uv run python -m stream_orchestrator +# Wait for `supervisor.session_ready` in logs. + +# Terminal 2 — showrunner (LLM planner + Playwright browser) +cd showrunner +uv sync +uv run playwright install chromium # one-time +export ANTHROPIC_API_KEY=... +uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --max-scenes 3 --timeout-sec 600 + +# (Optional) open the eye-check debug UI in a regular browser tab +open http://127.0.0.1:8765/debug +``` -# Terminal 2 — the autonomous, headed-Chrome showrunner -source .env && scripts/03-run-autonomous-show.sh +**OBS scene**: +- Browser source at `http://127.0.0.1:8765/` (the avatar, 1920×1080) +- Browser source at `http://127.0.0.1:8765/captions` (1920×220, position over avatar) +- Window Capture of the Playwright Chrome window the showrunner spawns (the main scene) -# Terminal 3 (optional) — Twitch chat ingester (Module 4) -# Viewers drop GitHub URLs; the avatar acknowledges them by name, navigates, -# and roasts. Requires SHOWRUNNER_EPISODE_MODE=true (already set if you sourced .env). -source .env && scripts/05-run-chat-ingester.sh +## What lives where -# Optional helpers -scripts/04-open-local-views.sh # open local viewer / debug pages -scripts/00-stop-local-processes.sh # clean up a stuck local process -``` +| Directory | Role | Runs as a process | +|---|---|---| +| [`stream_orchestrator/`](stream_orchestrator/) | M1 — HeyGen session lifecycle, hot-swap HLS rotation, the integration surface for everything else. Owns the OBS browser sources (`/`, `/captions`, `/activity`, `/debug`) and the `/producer/*` + `/activity/segments` HTTP contract. | ✅ yes | +| [`showrunner/`](showrunner/) | The content brain. LLM planner authors scenes (narration + browser action plan + cues); browser_executor performs them via Playwright; word_tracker fires cues on HeyGen's per-word timestamps so visuals land in sync with speech. | ✅ yes | +| [`script_producer/`](script_producer/) | Legacy minimal slice. Only 5 files (`word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`) — the showrunner imports these as a library via a sys.path hack in `showrunner/tools/*.py`. **No M2 daemon process runs.** | ❌ never started; library import only | -**OBS scene** — three sources pointed at the running M1: +## Docs -- Browser source `http://127.0.0.1:8765/` — the avatar (1920×1080) -- Browser source `http://127.0.0.1:8765/captions` — captions strip (1920×220, over the avatar) -- Window Capture of the Playwright Chrome window the showrunner spawns (the main view) +- **[TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md)** — start here. Architecture diagram, sync model (three clocks + the anchor that bridges them), end-to-end cue trace, current open problems P0-P5, file map, asks for help. Written for teammates jumping in. +- **[STATUS.md](STATUS.md)** — running engineering changelog. What changed when and why. +- **[showrunner/DEMO_RUNBOOK.md](showrunner/DEMO_RUNBOOK.md)** — showrunner-specific demo commands and SQLite event-log queries. +- **[stream_orchestrator/RUN.md](stream_orchestrator/RUN.md)** — M1 operational notes. +- **[stream_orchestrator/ARCHITECTURE.md](stream_orchestrator/ARCHITECTURE.md)** — M1 design doc. +- **[research/CLOUD_BROWSER_STRATEGY.md](research/CLOUD_BROWSER_STRATEGY.md)** — Bright Data / Browser Use Cloud review and how they should fit without breaking local media-clock sync. -For unattended 24/7 operation on a Mac Mini (launchd plists, system settings, OBS -heartbeat, a 7-day soak checklist), see -[docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md). +## Test suites -## Tests +Each package has its own venv (managed by `uv`). Current counts as of 2026-06-04: -Each package ships its own `pytest` suite. Run from inside the package directory: +```bash +cd stream_orchestrator && uv run pytest -q # 213 tests +cd ../showrunner && uv run pytest -q # 1362 tests (+7 deselected live tests) +cd ../chat_ingester && uv run pytest -q # 114 tests +cd ../script_producer && uv run pytest -q # 53 tests (subset of legacy suite) +# Total: 1,742 tests, all green +``` +Opt-in live tests (real Anthropic + real Chromium) run separately: ```bash -cd stream_orchestrator && uv run pytest -q -cd showrunner && uv run pytest -q -cd chat_ingester && uv run pytest -q -cd script_producer && uv run pytest -q +cd showrunner && uv run pytest -m live -q # 7 tests; requires ANTHROPIC_API_KEY ``` -The `showrunner/` suite additionally carries opt-in live tests (real Anthropic + -real Chromium) behind the `live` marker, deselected by default: +## Why the legacy `script_producer/` directory still exists -```bash -cd showrunner && uv run pytest -m live -q # requires ANTHROPIC_API_KEY +The showrunner needs `word_tracker.py` (~750 lines of marker-parsing + word-timestamp scheduling). That file lived in `script_producer/script_producer/` in the original repo. Rather than refactor it into the showrunner package, we kept it where it was and the showrunner imports it via: + +```python +# from showrunner/tools/run_autonomous_loop.py +_REPO_ROOT = Path(__file__).resolve().parents[2] +for _path in (_REPO_ROOT, _REPO_ROOT / "showrunner", _REPO_ROOT / "script_producer"): + if str(_path) not in sys.path: + sys.path.insert(0, str(_path)) + +from script_producer.word_tracker import WordTracker ``` -## Architecture +This standalone repo keeps that structure so the import works without changes. The only files in `script_producer/script_producer/` are the 5 that the showrunner actually imports. Everything else (the M2 daemon's tier_generator, Claude client, GitHub fetcher, etc.) was dropped. -| Directory | Role | Long-running process? | -|---|---|---| -| [`stream_orchestrator/`](stream_orchestrator/) | **M1** — HeyGen Avatar Realtime session lifecycle, hot-swap HLS rotation, and the OBS browser sources (`/`, `/captions`, `/debug`) plus the `/producer/*` and `/activity/segments` HTTP/SSE contract. | yes | -| [`showrunner/`](showrunner/) | The content brain. The LLM scene planner authors scenes (narration + browser action plan + cues); the browser executor performs them via Playwright; `word_tracker` fires cues on HeyGen's per-word timestamps so visuals land in sync with speech. | yes | -| [`chat_ingester/`](chat_ingester/) | Twitch chat reader → URL filter → SQLite queue the showrunner consumes. Pre-bakes per-message acknowledgements with Claude Haiku 4.5. | yes | -| [`script_producer/`](script_producer/) | Minimal library slice imported by the showrunner (`word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`). No standalone daemon runs. | no (import only) | - -Start with **[TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md)** — the architecture -diagram, the three-clocks sync model and the anchor that bridges them, an -end-to-end cue trace, and the current open problems. Other docs: - -- [stream_orchestrator/ARCHITECTURE.md](stream_orchestrator/ARCHITECTURE.md) — M1 design doc -- [stream_orchestrator/RUN.md](stream_orchestrator/RUN.md) — M1 operational notes -- [showrunner/DEMO_RUNBOOK.md](showrunner/DEMO_RUNBOOK.md) — showrunner demo commands and event-log queries -- [chat_ingester/README.md](chat_ingester/README.md) — chat pipeline details -- [HOST_CHARACTER.md](HOST_CHARACTER.md) — the host's persona and voice rules -- [docs/superpowers/specs/](docs/superpowers/specs/) — feature design specs -- [research/](research/) — browser-agent and cloud-browser research notes - -## License - -Released under the [MIT License](LICENSE). See [CONTRIBUTING.md](CONTRIBUTING.md) -to get set up for development. +**Cleanup that's NOT yet done**: properly fold `word_tracker.py` + `markers.py` into the `showrunner/` package, drop the sys.path hack, delete `script_producer/`. Mechanical work, ~1-2 hours. See `script_producer/script_producer/LEGACY_REPO_REVIEW_PROTOTYPE.md` in the original repo for the intent. + +## What was excluded from the extraction + +- **`activity_screen/`** (Module 3) — Playwright PNG screenshotter for the old `/activity` browser source. Unused in the showrunner architecture. If you need it back, it lives in the original `experiment-framework/activity_screen/`. +- **The HeyGen monorepo** (movio/, heygen/, etc.) — the Avatar Realtime v3 server source lives there; consult it directly for server-behavior debugging. +- **`ai-twitch-streamer/`** — a prior abandoned attempt at this same idea. Explicitly excluded. + +## Current state and known problems + +Episode v2 (one-LLM-call cinematic roasts) and §7+§8 (autonomous trending discovery + episode transitions + chat callbacks + link curiosity) are in. CostCap, Notifier, VerdictStore + leaderboard overlay, cross-session anti-repeat (Plan 8) are wired. Twitch chat ingester (Module 4) is built and waiting on env vars to engage. See [docs/superpowers/specs/](docs/superpowers/specs/) for the design history. + +Open work, in priority order: + +| Item | Status | +|---|---| +| Plan 9 — chat-pulse + @-mentions (LLM reacts to non-URL chat) | Spec written ([docs/superpowers/specs/2026-05-28-chat-pulse-and-mentions-design.md](docs/superpowers/specs/2026-05-28-chat-pulse-and-mentions-design.md)); not implemented | +| `word_tracker.prefix_timeout` | Legacy M2 code path; surfaced under load. Out of scope for showrunner-side fixes | +| OBS heartbeat → showrunner.notify consolidation | Documented in `.env.example` (point both webhook envs at same URL); real unification deferred | +| Long-soak live verification | Needs ≥1 hour live run on the Mac Mini setup | + +Tests pass (1,197 across all four suites). The known operational gotcha: chat ingestion requires `SHOWRUNNER_EPISODE_MODE=true` AND `CHAT_QUEUE_DB=state/chat_queue.sqlite` in the showrunner's env — without these, chat rows are written but never read. The `.env.example` template documents this contract. diff --git a/TECHNICAL_OVERVIEW.md b/TECHNICAL_OVERVIEW.md index 1f39ea1..fae70d4 100644 --- a/TECHNICAL_OVERVIEW.md +++ b/TECHNICAL_OVERVIEW.md @@ -2,7 +2,7 @@ 24/7 HeyGen avatar narrating a live Playwright browser show. Word-level timestamp sync via HeyGen's `/v3/avatar-realtime/{id}/words` SSE. -A standalone project. Everything runtime-required is in this directory. +Standalone extraction from a HeyGen monorepo. Everything runtime-required is in this directory. --- @@ -27,7 +27,13 @@ uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --ma open http://127.0.0.1:8765/debug ``` -**Test suites**: each package ships its own `pytest` suite. Run `uv run pytest -q` inside each subdir (`stream_orchestrator/`, `showrunner/`, `chat_ingester/`, `script_producer/`). The `showrunner/` suite additionally carries 7 opt-in `@pytest.mark.live` tests (deselected by default; require real API keys). +**Test suites** (run in each subdir): `uv run pytest -q` +- `stream_orchestrator/` → 167 tests +- `showrunner/` → 896 tests (+7 opt-in `@pytest.mark.live` deselected) +- `chat_ingester/` → 87 tests +- `script_producer/` → 47 tests (subset of original) + +**Total: 1,197 tests as of 2026-05-28.** **Sanity checks** while running: ```bash @@ -48,7 +54,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \ ``` ┌─────────────────────────────────────────┐ - │ HeyGen Avatar Realtime API │ + │ HeyGen Canary API │ │ (text in → audio + HLS video out) │ └─────┬───────────────────────────┬───────┘ ▲ │ @@ -95,7 +101,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \ | [`showrunner/`](showrunner/) | LLM scene planner → browser_executor (Playwright) + word_tracker (cue scheduler). | yes | | [`script_producer/`](script_producer/) | Library import only. 5 files used by showrunner: `word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`. Full M2 daemon was dropped. | no | -Not included: `activity_screen/` (M3 PNG screenshotter, unused), the full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), and HeyGen's backend service source. +Dropped from the extraction: `activity_screen/` (M3 PNG screenshotter, unused), full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), HeyGen monorepo source (`movio/`, `heygen/`, `lib/`). --- @@ -126,7 +132,7 @@ At `fire_wall_ms`, POST `/producer/marker_fired` → broadcast via `/activity/se **Why this works**: both `word.start` and `currentTime` are on the same axis. Anchor = the map to wall clock. Stalls freeze `currentTime`; re-anchor 250 ms later shifts `wall_t0` forward by the stall amount. -**Pipeline latency** (measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s). +**Pipeline latency** (canary, measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s). --- @@ -156,7 +162,7 @@ LLM authors `[CUE: {"op":"navigate","target":"https://github.com/foo/bar","lead_ | 1 | Emitter's `word.start` is accurate | ✅ HeyGen's responsibility, stable | | 2 | `