diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index ce374b5..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,687 +0,0 @@ -# Live Streamer Changelog - -## 2026-05-28 - -### Implemented (across multiple sub-sessions) - -**Episode v2 + §7+§8 transition + autonomous discovery (Plans 1-7 + transition spec)** -- Rich capture pipeline (`episode_capture.py`) — one hidden-browser snapshot per episode produces an `EvidenceBundle` (clean README + document-ordered outline + anchors + activity sidebar). -- Single-call episode author (`episode_author.py`) — one LLM call → ordered scenes + cue intents + verdict, both streaming and one-shot variants. `SHOW_VOICE` extracted into its own module; calibration ladder + bouncer persona. -- Deterministic grounder (`episode_grounder.py`) — cue intents → dense cues with the point-first auto-scroll model; visit_link cue grounded as deterministic click+back pair. -- Runtime wiring — `produce_episode` + `produce_episode_stream` chained into `autonomous_loop`; reactive fallback unchanged. -- Chat seeding (Module 4 → showrunner) — `chat_request_queue` consumed by `_select_next_episode_source` with phase-A walk + collapsed-callback prepending. -- Cross-episode memory (`ShowLedger`) — last 6 episode summaries rendered into the next author's `memory=` payload (Plan 6). -- Verdict persistence (`verdict_store.py` + leaderboard overlay) — per-repo verdicts written to `state/verdicts.sqlite` on verdict-scene serve, read by the existing `/overlay/leaderboard_data` endpoint (Plan 7). -- §7+§8 transition layer (`episode_transitions.py`) — four `TransitionKind`s (DIRECT_ACK / GOOGLE_ACK / CALLBACK / TRENDING_BROWSE), each with its own one-shot author prompt and template fallback. Autonomous trending picks rank-1 fresh repo (filtered against `ShowLedger.recent_slugs()`). -- Link curiosity (`link_peek.py`) — `httpx` pre-fetch on outline links matching `CURIOSITY_KEYWORDS`; up to 8 peeks per bundle. `visit_link` cue lets the avatar visit a demo/docs/playground page mid-scene and return. - -**Cross-session anti-repeat (Plan 8 — this session)** -- `VerdictStore.recent_summaries(n=6)` reads MAX(id)-per-repo for boot hydrate. -- `ShowLedger.hydrate(summaries)` — idempotent bulk-append, preserves newest-first order. -- `AutonomousLoop` gains `show_ledger=` kwarg; runner builds the hydrated ledger between `VerdictStore.open()` and the loop construction. A 6 AM bounce no longer re-roasts yesterday's repos. - -**Notifier → failure alarms** -- New `_dispatch_alarm_alert` helper on `AutonomousLoop`; `planner_failure_alarm` and `browser_failure_alarm` sites now also send structured alerts through the shared `Notifier` instance (Slack / Discord / macOS, rate-limited). - -**CostCap wiring** -- `AnthropicSceneClient` accepts `cost_cap=`; `complete()` and `complete_stream()` gate on `should_block()` before HTTP and record `usage.input_tokens + output_tokens` after success. A blocked call raises `ScenePlannerError("cost_cap_blocked")`, the failover planner takes over. -- Runner builds `CostCap.from_env() + Notifier.from_env()` once at boot and shares the instances across reactive, episode-author, and transition LLM clients. - -**Bug fixes** -- Transition flag lifecycle (`_episode_in_transitions`): no longer cleared in the stream's `finally:` block on success — must wait until visible URL matches `_episode_repo_url` (or a 90s safety timeout). Stops the cascade where the LLM finishes streaming ~30s into the avatar's ~90s narration and the divergence guard dropped the pick + roast scenes (verified live in the 08:01 run: 9 dropped-N events totalling 39 scenes). -- Verdict beat exempt from `EPISODE_MAX_ROAST_SCENES` cap (the verdict scene IS the spoken score; capping it drops the audio). -- TRENDING_BROWSE navigate cue anchored EARLY in the narration (`_find_early_anchor`) — late-anchored cues miss when word_tracker hits prefix_timeout. Mitigation for the freeze cascade. -- `_find_late_anchor` fallback returns ONE word not joined two — fixes the "HTML Sure" anchor-not-in-narration bug that dropped browse segments. -- Beat-uniqueness guard in both `author_episode` and `author_episode_stream` — drops a second `install` scene (verdict beat exempt). -- Trending-opener anti-repeat: banned-phrase list in the prompt + soft-warn observer (`episode.transition_banned_opener`). -- Per-cue-intent drop telemetry in the grounder (`episode.cue_intent_dropped reason=…`) — turns the silent `under_grounded` 0-cue scene failure mode into something grep-able. -- Visit_link: prompt rule relaxed from "≤1 per episode" to "1-2 per episode" with broader curiosity-keyword pool (27 keywords up from 11; cap raised 5→8). - -**Operations** -- `.env.example` — durable template documenting required `ANTHROPIC_API_KEY` + Twitch creds; opt-in vars for episode mode, chat queue, verdict store, cost cap, notifier, OBS heartbeat consolidation. **Loudly documents** that chat consumption requires `SHOWRUNNER_EPISODE_MODE=true` + `CHAT_QUEUE_DB=...` (the bot will write rows but the avatar will ignore them without both). -- `docs/operations/mac-mini-setup.md` — full unattended Mac Mini setup for 7-day operation: system settings, OBS scene composition, launchd plists (5 — M1, showrunner, chat ingester, OBS heartbeat, disk watchdog), Slack/Discord webhook wiring. -- New design specs in [docs/superpowers/specs/](docs/superpowers/specs/) — episode v2, transition+discovery, Plan 8 cross-session anti-repeat (executed), Plan 9 chat-pulse + @-mentions (deferred). - -### Verified - -- `showrunner`: full suite → **896 passing, 7 deselected** (opt-in live tests). -- `stream_orchestrator`: full suite → **167 passing**. -- `chat_ingester`: full suite → **87 passing**. -- `script_producer`: legacy timing-library subset → **47 passing**. -- **Total: 1,197 tests, all green.** - -### Open / deferred - -- Plan 9 chat-pulse + @-mention path — spec only, no code. -- `word_tracker.prefix_timeout` — surfaces under load. Showrunner-side mitigation (early-anchor navigate cue) is in; deeper fix is in the legacy `script_producer/word_tracker.py`. -- OBS heartbeat → `showrunner.notify` unification — documented in `.env.example` (point both webhook envs at same URL). -- Long-soak live verification (≥1 hour) on the Mac Mini setup. - ---- - -## 2026-05-25 - -### Implemented - -- Fixed headed live-tool layout so `run_autonomous_loop --headed --fullscreen` uses the requested OBS page viewport by default instead of inheriting macOS fullscreen's smaller native viewport. - - Default headed layout is now fixed `obs-1080p` page viewport. - - `--native-viewport` is the explicit escape hatch for real native viewport probing. - - Added `showrunner/tools/probe_browser_layout.py` for screenshot/viewport verification. -- Tightened LLM-authored scene budget from `43-50` words to a safer `42-49` planner target while keeping the hard `17-20s` segment contract at `2.5 words/sec` with a small estimate tolerance near the lower bound. -- Added pre-push planner repair for "too wordy but not technically over 20s" authored scenes, so `50+` word scenes get repaired before they reach HeyGen. -- Hardened visible target validation: - - `text=...` highlight/point/click/type cues are rejected during planning when the current viewport snapshot has elements and the target is not visible. - - `element:N` targets still resolve through the snapshot. - - visible substring text targets such as `text=Browser automation` are allowed when the snapshot contains `Browser automation headline`. -- Added regression coverage for the real failure shape where `text=clangd-lsp` existed somewhere in the DOM but had no visible box in the current viewport. -- Replaced stream-facing fallback narration that said "the planner tripped" with natural host language. Internal fallback reasons are now logged on fallback use and persisted on segment completion. -- Changed the live autonomous default to `max_in_flight_scenes=2` for smoother speech continuity. Operators can still force the conservative mode with `MAX_IN_FLIGHT_SCENES=1`. -- Added a safety guard for `max_in_flight_scenes=2`: unresolved page-changing handoffs block the next push until their browser cue is observed, so the avatar does not critique a hidden/preflight page before the visible browser has actually moved. -- Tightened the autonomous planner personality and repo-review contract: - - host voice is now specified as sharp late-night software critique with visible receipts, not neutral product narration, - - GitHub repo scenes must prioritize README/product substance, docs, install/quickstart, examples, homepage/demo links, and repair notes over stars, branches, tags, commits, sponsor buttons, nav tabs, and file-tree trivia, - - visible links now force visible `click` cues instead of lazy direct `navigate` cues when the destination is already on screen. -- Reweighted browser scout target extraction so GitHub README/docs/about/homepage/example/install/demo content outranks global GitHub chrome and metadata. -- Reset Chrome page zoom on browser start and after page changes/new tabs/searches to avoid persistent-profile zoom making the OBS browser look shifted or cropped. -- Added a two-pass scene repair ceiling for chained planner errors. Example: length repair can reveal metadata chatter, and the second repair now gives a stricter rewrite-from-scratch instruction instead of failing immediately. -- Narrowed the GitHub metadata gate so `tags` metadata is blocked, but normal wording such as "anchor tag" is allowed. -- Ran a real prompt-iteration pass against the autonomous planner contract: - - eval reports now count cue execution failures as failed turns instead of merely recording them, - - eval summaries include cue op histograms, scroll counts, and executed-cue success rate, - - planner payloads explicitly tell Claude that screenshots are visual evidence but image/screenshot text may not be selectable, - - director rejects PDF-like `navigate`, `navigate_address_bar`, and `new_tab` targets so arXiv PDFs do not trigger downloads in the stream browser. -- Rejected one prompt experiment after measurement: a soft "make it more human" anti-static instruction improved neither motion nor reliability and caused two real-turn failures. Kept the previous prompt instead of shipping vibes over evidence. - -### Verified - -- `showrunner`: `tests/test_authored_shots.py tests/test_llm_scene_planner.py` -> `43 passed`. -- `showrunner`: full suite -> `233 passed`. -- `showrunner`: fallback/autonomous/planner focused suite -> `44 passed`. -- `showrunner`: autonomous-loop/layout focused suite -> `25 passed`. -- `showrunner`: planner/director/executor/scout focused suite -> `70 passed`. -- Real LLM planner eval, no HeyGen, GitHub repo scenario, 2 turns, screenshot + viewport + cue execution: - - `ok_turns=2`, `failed_turns=0`, - - author latency mean `8.36s` (`5.91s` min, `10.80s` max), - - LLM-call latency mean `5.57s`, - - estimated speech `18.0s` and `19.2s`, - - host voice heuristic pass rate `1.0`. -- Real LLM planner eval, no HeyGen, five scenarios, 2 turns each, screenshot + page text + visible elements + scroll state + cue execution: - - `/tmp/showrunner_prompt_iter4_20260525/20260525T215729Z`, - - `ok_turns=10`, `failed_turns=0`, - - author latency mean `11.02s` (`5.65s` min, `13.88s` max), - - LLM-call latency mean `4.79s`, mode bucket `6.0s`, - - estimated speech mean `18.4s` (`16.8s` min, `19.6s` max), - - cue ops: `highlight=9`, `scroll=5`, `point=3`, `navigate=2`, `click=1`, - - executed browser cues: `20/20` succeeded. -- Failed prompt experiment, same five-scenario eval: - - `/tmp/showrunner_prompt_iter5_20260525/20260525T220226Z`, - - `ok_turns=8`, `failed_turns=2`, - - failures were too-short scene repair and invalid post-navigation click ordering, - - experiment reverted. -- `git diff --check` -> passed. -- Headed layout probe: - - fixed OBS viewport: configured `1920x1080`, executor viewport `1920x1080`, screenshot `1920x1080`, - - native fullscreen probe reproduced the bad macOS viewport shape: executor viewport `1496x776`, screenshot `2992x1552`. - -## 2026-05-24 - -### Implemented - -- Increased the active runway baseline from `20s` to `30s`: - - showrunner push gate defaults, - - direct-producer `max_runway_sec` defaults, - - run scripts, - - deterministic runway proof. -- Lengthened the startup seed to 112 words, about `44.8s` at `2.5 words/sec`, so the first real Claude scene has roughly the same 15s planning margin before the 30s gate opens. -- Fixed the real continuity bottleneck: the planner may now prepare the next scene while one pushed scene is still in flight. `max_in_flight_scenes=1` still limits what has already been sent to HeyGen; it no longer prevents Claude from drafting the next near-term scene. -- Split scout navigation into two explicit modes: - - default `shadow_preflight`: scout chooses the URL, a hidden browser loads/screenshots it for Claude, and OBS-visible Chrome stays put until the avatar speaks a timed `navigate` cue, - - `timed_handoff`: same visible timing behavior, but without hidden preflight analysis, - - `immediate_observe`: the older eval/development behavior that navigates before narration so Claude can author from the destination page immediately. -- Added page-handoff preplanning safety: if an in-flight scene contains a page-changing cue, the planner waits until that cue resolves before preparing the next scene. Non-page-changing scenes can still preplan ahead for voice continuity. -- Added a real shadow-preflight smoke with saved screenshots. It proves the OBS-visible browser remains unchanged during hidden preflight, while Claude receives the hidden target page and screenshot before the timed navigate cue moves the visible stage. -- Removed legacy producer-idle/silence fields from active M1 debug/runway payloads and the debug UI. Runway is the only active hot-path push gate. -- Browser cursor polish: - - page-changing clicks now detect URL changes and settle the cursor after navigation, - - navigate/back/forward/new-tab/close-tab/switch-tab/address-bar flows now land the cursor in the central viewing rectangle, - - page-change wander is clamped inside that central rectangle instead of drifting toward dead corners. -- Debug UI now separately shows browser action count and browser error count from showrunner events, so click/navigate/highlight execution is easier to verify alongside cue marker fires. -- Fixed direct-showrunner runway truth: `/producer/runway`, `/debug/state`, and `/producer/push_segment` now prefer the wrapper/player `video.currentTime` sample when the player clock is fresh. During HLS stalls, runway stays pinned to the frozen media clock instead of draining by wall time. -- Added `player_clock_ready` / `runway_clock_source` telemetry. Showrunner pushes now wait while the player clock is missing/stale, so cue-bearing scenes are not sent before `word_tracker` can map words to the OBS-visible media clock. -- Hardened `word_tracker` cue firing to use sampled `video.currentTime` when available and reject player-clock anchors older than `2s`; stale anchors no longer fire browser commands by wall-clock extrapolation. -- Expanded the LLM planner contract so Claude knows the current cue vocabulary: wait/wander durations, new-tab URL behavior, switch-tab targets, back/forward/close-tab, keyboard ops, `google_search`, and indexed `element:N` targets. - -### Verified - -- `showrunner`: full suite -> `219 passed`. -- `stream_orchestrator`: full suite -> `135 passed`. -- `script_producer`: full suite -> `46 passed`. -- `script_producer`: `tests/test_word_tracker_timing.py` -> `23 passed`. -- `tools.probe_sync_and_runway` -> `ok: true`; `30.01s` runway refuses push, `30.0s` accepts push, and browser cue firing still waits for player-clock truth. -- Focused BrowserExecutor smoke confirmed both click-navigation and direct navigation leave the visible cursor inside the central viewing rectangle. -- `debug.html` inline JavaScript passes `node --check`. -- Shadow-preflight screenshot smoke: - - visible before and visible after preflight are byte-identical, - - hidden preflight loaded the target page, - - planner received the target page URL/title plus a `24,219` byte screenshot, - - simulated timed navigate cue then moved the visible browser to the target page. - -## 2026-05-23 - -### Implemented - -- Added operator run scripts: - - `scripts/01-login-browser-profile.sh` for manual Google login with the dedicated persistent Chrome profile, - - `scripts/02-start-avatar.sh` for M1/external producer mode, - - `scripts/03-run-autonomous-show.sh` for the real headed autonomous loop with current pacing defaults, - - `scripts/04-open-local-views.sh` for local viewer/debug pages, - - `scripts/00-stop-local-processes.sh` for stuck local process cleanup. -- Added `research/CLOUD_BROWSER_STRATEGY.md` after reviewing Bright Data Agent Web Access / Browser API and Browser Use Cloud. Decision: copy the remote-CDP/profile/proxy/live-preview strategy later, but keep local headed Chrome as the default v1 OBS-visible runtime until remote CDP proves media-clock sync and capture quality. -- Refined browser-command polish from operator feedback: - - scroll no longer moves the cursor into the center if the cursor is already inside the central reading rectangle, - - Google search no longer parks the cursor in the bottom-left after results load; it moves to a natural viewing posture on the results page, - - default cursor wander increased from `7px/2 steps` to `10px/3 steps`, - - the lab now has separate `New Blank Tab` and `New Tab URL` buttons so "new fresh tab" is not confused with opening the URL field in a new tab, - - `BrowserExecutorConfig.user_data_dir` plus `SHOWRUNNER_BROWSER_USER_DATA_DIR` / `--browser-user-data-dir` allow an opt-in dedicated Chrome profile for persisted Google sign-in. -- Updated the current dedicated-queue pacing baseline: - - `seed_text` is now 112 words, about `44.8s` at 2.5 words/sec. - - Disabled bootstrap continuation text. - - `max_silent_sec` is now `0` and is a no-op in M1 pacing. - - Removed the active showrunner producer-silence force-push path. - - Default push gate is now `30s`. - - Default max in-flight scenes is now `1`. - - Scene target is now `17-20s`, about `43-50` words at `2.5 words/sec`. - - Segment compiler hard cap is now `20s`; the LLM planner rejects below `17s` and asks once for repair. - - Claude temperature default is now `0.5`. - - Browser navigation/action Playwright timeouts default to `0`, meaning no client-side timeout. -- Updated deterministic browser-director fallback narration so non-authored fallback scenes pad into the 43-50 word band instead of producing tiny chunks. -- Disabled fake-success sync fallbacks in `WordTracker`: - - no fixed wall-clock Path B when no player clock exists, - - no tail-timeout firing for remaining cues, - - no player-clock stall/absolute fallback fire. -- If a word timestamp exists but the player anchor has not arrived yet, the cue now waits for a matching player clock. If the word timestamp itself is missing, the cue is logged as unmappable and is not fired. -- Added `showrunner/tools/eval_llm_scene_planner.py`, a real-browser LLM eval harness that mirrors the autonomous loop's scout -> navigate -> observe -> screenshot -> author path and stores sanitized raw requests, raw responses, latency, parsed scene, scores, and optional cue-execution screenshots. -- Added `showrunner/tools/browser_command_lab.py`, a headed Chrome command lab with a local control UI, built-in fixture page, motion knobs, visible target suggestions, action log, screenshots, and buttons for every current browser cue op. -- Added explicit browser-lab operator controls to relaunch headed Chrome and bring Google Chrome to the front, because the previous lab started Chrome implicitly and made the controlled browser easy to miss. -- Improved browser action realism after operator QA: - - scroll now uses the actual native Chrome viewport height, central cursor posture, smaller `small_up/small_down` variants, and eased wheel deltas instead of constant-speed chunking, - - scroll keeps the normal cursor icon instead of pretending the page is being grabbed, - - cursor actions now include configurable idle wander/micro-movement and a larger explicit `wander` action, - - highlight now performs a visible drag-style selection before applying DOM/native selection, - - typing moves to the field, clicks it, then types with per-character delay, - - page snapshot targets are deduped by visible text and include indexes, selectors, geometry, and affordances such as `type`, `click`, `navigate`, and `read`, - - lab config inputs no longer reset while the operator is editing them, - - added manual keyboard commands: Enter, Backspace, Delete, hotkey, wander, URL shortcut/go, and `google_search`. -- Fixed confusing lab command routing: - - `URL Shortcut / Go` now uses only the URL field and rejects DOM targets such as `text=Jump to deep section`, - - `New Tab URL` now uses URL/Value instead of the selected DOM target, - - `Switch Tab` now uses the Value field or clickable tab pills, and the executor calls `bring_to_front()` so the visible Chrome tab actually changes. -- Added first-class `google_search` browser cue. It opens Google in a new tab, types the query into the search box when possible, and lands on the result page. Planner instructions treat it as a final handoff; the next scene must observe the results before narrating them. -- Reviewed Browser Use and OxyMouse implementation patterns: - - copied the Browser Use idea of indexed, affordance-rich browser state for LLM clarity, - - copied Browser Use's scroll-context idea by adding page-depth telemetry to snapshots and the LLM payload, - - kept our synced cue executor instead of adopting Browser Use as the runtime because Browser Use does not know about avatar media-clock cue timing, - - copied OxyMouse-style ideas: Bezier paths, acceleration/deceleration phases, corrective micro-movements, and random movement testing; did not add the dependency yet. -- Downloaded durable local references outside the repo: - - `/Users//browser-agent-references/browser-use` - - `/Users//browser-agent-references/OxyMouse` -- Added `showrunner/showrunner/movement_model.py`, a dependency-free movement layer inspired by OxyMouse phases: - - `cursor_path()` for accelerated/cruising/decelerating/corrective cursor travel, - - `wheel_deltas()` for eased scroll wheel events, - - `wander_targets()` for visible browsing drift. -- Added `research/BROWSER_AGENT_PATTERNS.md` with the local Browser Use/OxyMouse review and decisions. -- Hardened the browser overlay so cursor/scroll/highlight actions re-inject the overlay after page loads instead of failing when `window.__showrunnerOverlay` disappears. -- Added first-class `element:N` / `index:N` / `#N` viewport-element targets: - - `snapshot_page()` now includes `scroll_state` with pages above/below and vertical page percentage, - - the planner prompt now prefers `element:N` for visible point/highlight/click/type cues and receives `scroll_state`, - - the browser director resolves LLM-authored indexed targets before safety validation, - - the browser executor resolves indexed targets at runtime for manual lab/API cues, - - the command lab target suggestions now set `element:N` and show page-depth telemetry. -- Added `showrunner/tools/probe_sync_and_runway.py`, a deterministic developer proof for the two hard gates: runway-only pushing and player-clock-only cue firing. -- Tightened planner prompt control after real evals: - - review modes must include one concrete improvement beat, - - authored shots now have explicit per-shot word ceilings, - - length-repair prompts now target exactly 2 balanced shots and `48-50` spoken words. -- Bounded `LLMScenePlanner` raw message history to the latest `8` messages by default. Long-term continuity should come through compact `PromptMemory`, not unbounded raw JSON replay. -- Fixed authored page-handoff validation to compare page-changing cues against the actual observed browser snapshot URL instead of the model's top-level requested URL. -- Added one-shot planner repair for page-changing handoff errors and malformed/targetless authored cues, not only overlong/short segment length. - -### Verified - -- `showrunner`: `tests/test_segment_plan.py tests/test_runtime.py tests/test_llm_scene_planner.py tests/test_autonomous_loop.py tests/test_browser_director.py` -> `71 passed` -- `stream_orchestrator`: `tests/test_pacing.py tests/test_feeder.py tests/test_config.py tests/test_direct_producer.py` -> `42 passed` -- `script_producer`: `tests/test_word_tracker_timing.py` -> `22 passed` -- `showrunner`: `tests/test_llm_scene_planner.py` -> `18 passed` -- `showrunner`: `tests/test_llm_scene_planner.py tests/test_authored_shots.py tests/test_browser_director.py` -> `51 passed` -- `showrunner`: `tools.probe_sync_and_runway` -> `ok: true` -- Browser command lab fixture API smoke: - - point, click, type, highlight, scroll up/down, anchor click, back, forward, new tab, switch tab, close tab, wait, screenshot all returned successful action records. - - post-QA smoke: config edit, small smooth scroll with default cursor, drag-style highlight, click-before-type, larger wander, invalid URL rejection, new tab, visible switch tab, and Google search all returned expected action records. - - movement-model smoke: small scroll, wander, and click all returned successful action records after restarting the headed lab. -- `showrunner`: `tests/test_movement_model.py tests/test_bezier_path.py tests/test_browser_executor.py tests/test_browser_scout.py tests/test_curiosity_ops.py tests/test_authored_shots.py tests/test_llm_scene_planner.py` -> `76 passed` -- `showrunner`: `tests/test_browser_targets.py tests/test_browser_scout.py tests/test_authored_shots.py tests/test_llm_scene_planner.py tests/test_browser_executor.py` -> `62 passed` -- `showrunner`: `tests/test_movement_model.py tests/test_bezier_path.py tests/test_browser_executor.py tests/test_browser_scout.py tests/test_curiosity_ops.py tests/test_cues.py tests/test_browser_pipeline.py tests/test_authored_shots.py tests/test_browser_director.py tests/test_llm_scene_planner.py` -> `104 passed` -- Browser command lab index/scroll smoke: - - `point element:4` resolved to the fixture button, - - `type element:1` resolved to the fixture input, - - `scroll down` moved the viewport from top-of-page to about `62.2%` depth and reported updated `pages_above/pages_below`. -- Updated stale buffer/scroll tests to match the current runway-only and smoother-scroll behavior. -- Tightened page-changing handoff enforcement: - - clicks on visible elements with `navigate` affordance or `href` are now treated as page/session handoffs, - - link-like clicks must be the final authored shot unless a later planner pass has observed the new page, - - this prevents the avatar from clicking into a new file/page and immediately narrating unobserved content. -- Real LLM Browser Use-style contract eval, no HeyGen: - - `/tmp/showrunner_llm_evals_index_contract/20260524T054613Z`: `3/3` valid; author turn mean `9425.99ms`, all Claude-call mean `4020.57ms`, vision author-call mean `4711.61ms`. - - `/tmp/showrunner_llm_evals_handoff_fix/20260524T055240Z`: GitHub Trending rerun valid `17.6s`; final cue is the `CLAUDE.md` click handoff, with no post-click unobserved scroll. -- Autonomous-loop smoke, no HeyGen: - - `tools.probe_sync_and_runway` -> `ok: true`, with no push above `30.0s`, push at `30.0s`, and no browser cue fire before player clock. - - `tools.smoke_autonomous_loop_fake` -> 3 browser actions / 3 marker events, runtime chose `draft_more`, `commit_from_draft`, `push_to_heygen`. - - `tools.smoke_autonomous_loop_multi_fake` -> `ok: true`, 3 scenes completed, 7 browser actions, 7 marker events. -- Real HeyGen one-scene run: - - Session `` warmed to streaming in `12.34s`. - - Wrapper player clock was healthy and cues fired through `A-player`. - - Marker alignment drift: about `45.8ms`, `42.3ms`, and `39.6ms`. - - Planner fell back to the old demo fake scene after primary overlong/repair failure; that pushed invalid demo targets onto GitHub Trending. - - Cleanup posted `final=true`; no terminal status inside 5s; `/cancel` returned `cancelled=true`. -- Replaced real-run deterministic fallback with `ViewportSafeFallbackSceneClient`: - - no page-specific fake selectors, - - only `wait` plus `scroll` on the current page, - - generic narration that buys time without inventing receipts. -- `showrunner`: full suite -> `211 passed` -- Browser-command polish verification: - - focused executor/movement tests -> `20 passed`, - - persistent-profile startup smoke passed with a temporary profile, - - command-lab API smoke confirmed `New Blank Tab` opens `about:blank`, - - command-lab API smoke confirmed Google search cursor lands near the results area instead of bottom-left, - - command-lab API smoke confirmed scroll does not move the cursor when it starts inside the central reading rectangle. -- `showrunner`: full suite -> `212 passed` -- Real LLM evals, no HeyGen: - - scout path `/tmp/showrunner_llm_evals_real/20260524T021015Z`: valid `19.2s` / `48` words, scroll + highlight executed successfully, authoring path `10008.48ms`. - - no-scout path `/tmp/showrunner_llm_evals_real/20260524T015809Z`: valid `20.0s` / `52` words, scroll + highlight executed successfully, authoring path `10351.4ms`. - - four-scenario batch `/tmp/showrunner_llm_evals_real_batch/20260524T032348Z`: `3/4` valid before the handoff/repair fixes; all Claude calls mean `4315.96ms`, mode bucket `2000ms`; vision author calls mean `5618.01ms`, mode bucket `4000ms`. - - post-fix Wikipedia rerun `/tmp/showrunner_llm_evals_real_batch/20260524T033002Z`: valid `18.8s`; all Claude calls mean `4267.17ms`, mode bucket `2000ms`; vision author calls mean `5188.6ms`, mode bucket `4000ms`. - -### Still Pending - -- Run one bounded real HeyGen E2E with the new ~45s seed / 30s runway gate and confirm cleanup reaches terminal status or `/cancel`. -- Ask HeyGen whether a single realtime HLS session's `video.currentTime` is expected to be monotonic and how `/words` timestamps map across playlist discontinuities/currentTime resets. -- Add a short media-only validation path with wrapper watchdog disabled/enabled before another full LLM+HeyGen burn. -- Add startup preplanning before stream create if the ~45s seed is still not enough. -- Cue schema unification between `showrunner` and `script_producer`. -- Visual QA screenshot assertions for real headed Chrome cursor/selection/click/scroll polish. -- Real LLM tone pass under the new 17-20s scene band. - -### Historical Notes From Earlier 2026-05-23 Work - -- Tuned the responsive pacing baseline: - - Shortened the startup seed/continuation text so the first real scene does not sit behind a long canned monologue. - - Lowered default autonomous push gate to `14s`, polling to `0.5s`, scene target to 8-12s, and hard segment cap to `16s`. - - Added `--heygen-push-below-sec`, `--push-poll-interval-sec`, and `--max-in-flight-scenes` to `tools.run_autonomous_loop`. - - Added planner timing logs so real runs show how long Claude/browser planning took. -- Added one-shot overlong-scene repair in `LLMScenePlanner`. - - If Claude returns an otherwise valid scene that exceeds the hard speech cap, the planner asks once for corrected JSON at 24-30 words. - - Failed overlong attempts are still not appended to conversation history. -- Added primary-player clock hygiene: - - `wrapper.html` now generates a `client_id` for playback anchors and supports `?clock=off`. - - `SessionClock` elects the first fresh player as primary and ignores side-player anchors until the primary goes stale. - - `tools.measure_media_playback` defaults to `--clock-mode off` and now writes a summary on SIGINT/SIGTERM. -- Reviewed the copied HeyGen reference update (internal backend fix) under `api_references/`. - - Confirmed the idle-timeout change is server-side `avatar_realtime_runtime_config`, not a new create-session request field. - - Confirmed the normal clean finish path remains `/text` with `final=true`. - - Confirmed the new hard-abort path is `POST /v3/avatar-realtime/{stream_id}/cancel`. -- Added M1 support for the new `/cancel` endpoint. - - `ApiClient.cancel_session()` calls the endpoint and returns the `cancelled` boolean. - - `Session.cancel()` exposes the hard-abort path. - - Supervisor teardown still prefers `final=true`, then falls back to `/cancel` if final cannot be posted. -- Added `stream_orchestrator/tools/probe_idle_timeout.py`. - - Creates a real text-stream session, waits without extra text, and reports whether the session survives past a threshold. - - This is the one-command proof to run after HeyGen applies the long `idle_timeout_sec` runtime config. -- Added the default two-phase autonomous planning path: - - scout a safe public URL/topic first, - - navigate the real browser, - - observe the actual page, - - then author narration/cues from visible state. -- Added `--one-shot-planner` as an escape hatch; default runs two-phase. -- Added restrained host-personality guidance to the planner: dry, observant, useful, bounded curiosity. -- Added `HOST_CHARACTER.md` as the durable show bible for the avatar's character, roast rules, show formats, ratings, and guardrails. -- Upgraded the planner's runtime personality from generic "dry host" to Live Critic Desk: - - evidence-first software satire, - - Repo Roast Court / Landing Page Smell Test / Docs Interrogation / Demo Theater Watch / Repair Minute modes, - - artifact-focused roasts with one useful repair note, - - optional ratings such as `README Wearing A Blazer` and `Grandma With Claude Code Could Ship This`. -- Changed the default autonomous-loop show goal from public-web research to public-web software critique. -- Tightened authored-shot prompt rules after a real LLM smoke emitted a targetless `highlight` cue; the planner now explicitly says targetless authored cues are rejected. -- Added `tools.smoke_browser_scene_matrix` to run real LLM browser-scene QA across multiple public websites and save viewport screenshots after each cue. -- Tightened browser-scene reliability from the matrix findings: - - visible text targets now prefer a visible match instead of the first offscreen match, - - `highlight` can auto-scroll the target into view when the planner slightly undershoots, - - planner scenes are compiled/length-validated before history is appended, - - authored anchors fall back to the first words of the shot when the model supplies a bad anchor, - - positive authored `lead_ms` values are normalized to safe non-positive defaults, - - page-changing cues are now treated as handoffs to the next observed scene. -- Enforced explicit page-changing authored cues as final-shot handoffs in the director, so the next planner turn must observe the new page before narrating details from it. -- Shortened LLM scene target length to 10-16 seconds / under 42 words and lowered the hard segment cap to 22 seconds so the stream reacts faster and avoids stale buffers. -- Added immediate speech-tracking failure recovery: - - `WordTracker` now keeps pushed segments as a FIFO, - - reports active and pending segments abandoned on session swap, - - the autonomous loop turns those reports into immediate scene failures instead of waiting for the full browser-action timeout. -- Added prompt/source-choice refinements from the transcript pass: - - avoid random catchphrases, - - use setup -> receipt -> joke -> repair note, - - avoid Hugging Face model pages for v1 unless already observed as public/readable, - - keep quotes short and visible. -- Added a selector reliability guard for authored highlight/point cues: - - empty targets are rejected, - - long unstructured highlight targets are rejected, - - planner prompt now prefers exact selectors or short exact visible-element text. -- Added safe voice-control tag handling: - - `spoken_text` keeps tags such as `[sighs]` for the voice engine, - - `word_tracking_text` strips them for cue alignment, - - `WordTracker` ignores bracket/SSML control tokens if they appear in `/words`. -- Kept captions aligned with that same policy: captions filter the broader voice-control token set, and the caption typography now uses zero letter spacing. -- Added operator debug visibility: - - M1 now accepts `POST /debug/showrunner_event` and includes recent showrunner debug events in `/debug/state`. - - `tools.run_autonomous_loop` mirrors sanitized Claude requests, raw Claude responses, loop config, and browser executor outcomes into the M1 debug page. - - `BrowserPipeline` has an optional debug callback for action/error outcomes. - - `/debug` now displays last Claude request, raw response, latency, last pushed HeyGen text/markers, cue-fire stream, word stream, media drift, and last browser executor outcome. -- Updated root `PLAN.md` with the current working/not-working state, exact runtime constants, media-continuity blocker, and current debug UI contract. - -### Verified - -- Responsive pacing patch checks: - - `showrunner/tests/test_llm_scene_planner.py tests/test_autonomous_loop.py tests/test_media_telemetry.py` -> `39 passed` - - `stream_orchestrator/tests/test_session_clock.py tests/test_direct_producer.py tests/test_captions_page.py` -> `39 passed` - - `tools.measure_media_playback --help` verified the new `--clock-mode` flag. - - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - Real Anthropic/browser smoke without HeyGen passed on GitHub Trending; it produced a short two-shot scene with a final page-changing handoff. -- Latest dedicated-queue real E2E sample: - - First request warmup was `43.16s`. - - First real content push had runway `-1.09s -> 14.37s`. - - Loop stopped with `planner_failure_alarm` because Claude returned several 17-25s scenes against the new 16s hard cap. - - Media summary showed median lag `5.38s`, mean `7.30s`, one HLS currentTime reset, five stalls, max stall `11.65s`. - - Marker drift for fired markers remained good: mean `8.78ms`, max `17.57ms`. - - The old stream `` is confirmed `completed` with `end_reason=final_marker`; no cancel was needed. -- Second dedicated-queue validation run: - - Started showrunner earlier and reduced in-flight scenes to 1 to test startup pre-planning. - - Warmup was `9.91s`. - - First content push still arrived late: runway `-7.85s -> 7.56s`. - - Media summary was bad: `continuous=false`, mean lag `42.04s`, median lag `41.68s`, max lag `76.34s`. - - HLS had 4 backward jumps to `0`, 13 stalls, `79.61s` total stall time, max stall `12.92s`, observed playback rate `0.452x`. - - Marker paths were mixed: 1 normal `A-player`, 1 `A-player-fallback`, 1 `tail-timeout`. - - Conclusion: stop tuning lower than 14s until HLS continuity / `/words` timeline mapping is clarified. -- Debug visibility patch checks: - - `stream_orchestrator/tests/test_debug_page.py tests/test_direct_producer.py` -> `20 passed` - - `showrunner/tests/test_browser_pipeline.py tests/test_llm_scene_planner.py tests/test_autonomous_loop.py` -> `43 passed` - - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`, proving the debug bridge fails silent when no M1 debug receiver is available. -- Focused post-audio-tag tests: - - `showrunner/tests/test_segment_plan.py tests/test_producer_client.py` -> `14 passed` - - `script_producer/tests/test_word_tracker_timing.py` -> `12 passed` -- Latest full suite baseline from this patch series: - - `showrunner`: `178 passed` - - `stream_orchestrator`: `127 passed` - - `script_producer`: `38 passed` -- Post-(internal backend fix) integration checks: - - `stream_orchestrator/tests/test_client.py tests/test_supervisor_recovery.py` -> `6 passed` - - `uv run python -m tools.probe_idle_timeout --help` verified. - - `stream_orchestrator` full suite -> `127 passed` - - `showrunner` full suite -> `178 passed` - - `script_producer` full suite -> `38 passed` -- Latest local browser smokes from this patch series: - - `tools.smoke_browser_executor` passed. - - `tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. -- Real Anthropic planner smokes: - - `tools.smoke_llm_scene_planner --real` passed. - - `tools.smoke_llm_scene_planner --real --scout-first` passed and returned a safe Playwright docs scout URL. - - `tools.smoke_llm_scene_planner --real --scout-first --browser-observe --headed` passed: scouted Playwright docs, opened headed Chrome, observed the real page, and authored accepted cues from that snapshot. - - Re-ran `tools.smoke_llm_scene_planner --real --scout-first --browser-observe --headed` after the Live Critic Desk prompt update. It chose `https://github.com/microsoft/playwright`, opened the real repo, produced a Repo Roast Court segment, praised the README when the evidence was good, and authored valid scroll/highlight cues for visible README text. - - Re-ran after the personality transcript pass. It again chose `https://github.com/microsoft/playwright`, authored valid scroll/highlight cues, and produced a specific evidence-first segment at `18.97s` / `50 words`. Tone is usable; length still needs tightening toward 35-42 words. -- Real headed Chrome browser-scene matrix: - - `tools.smoke_browser_scene_matrix --real-llm --headed` passed across `github_trending`, `github_repo`, `github_click_file`, `docs_quickstart`, `wikipedia`, and `arxiv`. - - Covered navigate, scroll, highlight, auto-scroll-to-highlight, and a real visible click on `CLAUDE.md`. - - Cursor overlay was visible for every recorded action. - - Viewport screenshots are in `/tmp/showrunner_scene_matrix`. -- Real HeyGen canary smoke: - - Started `stream_orchestrator` against `api.heygen.com`. - - Ran `tools.smoke_real_loop --timeout-sec 180`. - - Result: `typed="real canary loop"`, `browser_actions=3`, `marker_events=3`, pushed after `67.42s`. - - Marker fire path was `A-player` for all 3 cues with drift about `0.3ms`, `9.0ms`, and `33.1ms`. - - Inspected `/tmp/showrunner_real_loop.png`; typed text and native text-selection highlight were visible. - - Stopped the orchestrator after the smoke; port 8765 was confirmed closed. -- Real headed autonomous HeyGen run after FIFO/session-ready patches: - - Command: `tools.run_autonomous_loop --real-llm --headed --layout-preset obs-1080p --autonomy-mode curious --max-scenes 3 --timeout-sec 480 --producer-silence-force-push-sec 20` - - Result: `ok=false`, `stop_reason=browser_failure_alarm`, `scenes_pushed=3`, `scenes_completed=0`, `marker_events_total=0`, `browser_actions_total=0`. - - Browser navigation itself worked; final screenshot was `/tmp/showrunner_e2e_final.png`. - - The current canary environment still rotated sessions from `idle_timeout`, causing `WordTracker` to abandon active/pending segments before markers fired. - - Conclusion at the time: the long-running demo was blocked on server-side queue/idle config, not on headed Chrome. -- HeyGen idle probe (after the internal backend fix): - - The internal backend fix was confirmed merged. - - Ran `STREAM_ORCHESTRATOR_EXTERNAL_PRODUCER_MODE=1 uv run python -m stream_orchestrator` against canary with no showrunner pushes. - - Canary still returned `completed/end_reason=idle_timeout` after about `36.19s` and again after about `38.37s`. - - Clean Ctrl-C on the third session posted `final=true` successfully with `byte_drift=0`; `/cancel` fallback was not needed in that shutdown. - - Conclusion: the code is merged, but this account/space still needs the server-side `avatar_realtime_runtime_config` change, likely `idle_timeout_sec: 86400` or another high positive value. -- HeyGen long-idle confirmation after the dev applied the per-space config: - - `uv run python -m tools.probe_idle_timeout --min-survival-sec 90` - - Result: `ok=true`, `reason=survived_minimum_without_idle_timeout`, `status=streaming`, `end_reason=null`. - - The session survived the old 36-38s failure window and the 90s probe window; cleanup posted `final=true`. - - Production showrunner runs should use runway-only pacing with `--producer-silence-force-push-sec 0`. -- Headed Chrome viewport fix: - - Confirmed the old headed launch reported `innerWidth=1920`, `innerHeight=1080` while the real Chrome window was only about `1476x843`, so Playwright screenshots were not trustworthy for OBS fit. - - Changed headed `BrowserExecutor` to use Chrome's native visible viewport by default. - - Changed `BrowserExecutor.screenshot()` to viewport-only so visual QA shows what viewers see, not the whole page. - - Added `--emulated-viewport` as an explicit escape hatch for fixed Playwright viewport geometry. - - Verified patched headed Chrome reports `innerWidth=1496`, `innerHeight=776`, and `BrowserExecutor.viewport_size()` returns `1496x776` on this Mac display. - - Focused tests: `showrunner/tests/test_browser_executor.py tests/test_director_polish.py` -> `27 passed`. -- Added stale avatar-player clock filtering: - - An old wrapper tab was still posting `/control/video_started` anchors for a previous HeyGen session id after orchestrator restart. - - M1 now ignores player-clock anchors unless the posted `session_id` matches the current active session. - - Focused tests: `stream_orchestrator/tests/test_direct_producer.py tests/test_client.py tests/test_supervisor_recovery.py` -> `16 passed`. -- Updated HeyGen runtime assumption from "unlimited idle" to the dev-confirmed dedicated queue contract: - - Dedicated queue is enabled for this space. - - No-audio kill window is 30 minutes. - - There is only one host, so every restart must release the current stream first. - - If dedicated queue produces corrupted video, ask HeyGen to flip `use_dedicated_queue=false` for this space. -- Tightened teardown for the dedicated-host reality: - - M1 now posts `final=true`, waits briefly for terminal status, and calls `/cancel` if the session still reports `streaming`. - - This covers the observed behavior where `final=true` was accepted but status remained `streaming` for several seconds. - - Focused tests: `stream_orchestrator/tests/test_supervisor_recovery.py tests/test_direct_producer.py tests/test_client.py` -> `17 passed`. - -### Historical Pending Before Latest Pacing Patch - -Superseded by the current `Still Pending` list at the top of this date section. - -## 2026-05-22 - -### Added - -- Created this root `CHANGELOG.md` as the durable handoff log for Codex and Claude Code. -- Created root `PLAN.md` with the current architecture, findings, sync contract, demo direction, and next patch list. - -### Audited - -- Reviewed current repo modules while avoiding `ai-twitch-streamer`. -- Reviewed: - - `stream_orchestrator` - - `showrunner` - - `script_producer` - - `api_references` - - root docs: `README.md`, `STATUS.md`, `HANDOFF.md`, `TECHNICAL_OVERVIEW.md` -- Reviewed the local Poke files supplied from Downloads and summarized usable personality principles without copying prompt text into the app plan. -- Reviewed current browser-control ecosystem sources: Browser Use, Browserbase, Agent Browser, BrowserMCP, Playwright, Selenium Actions, OxyMouse, HumanCursor, DeepMind AI Pointer, VedalAI Neuro SDK, and Excalidraw MCP. - -### Verified - -- Unit tests: - - `stream_orchestrator`: `uv run pytest -q` -> `123 passed in 68.81s` - - `showrunner`: `uv run pytest -q` -> `157 passed in 1.28s` - - `script_producer`: `uv run pytest -q` -> `34 passed in 0.34s` -- Fake/local smokes: - - `showrunner/tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - `showrunner/tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - `showrunner/tools.smoke_browser_executor` passed all local actions: navigate, highlight, point, type, click, scroll, highlight. - -### Findings - -- P0: The stream can still hit HeyGen `idle_timeout` in external producer mode. - - `showrunner/showrunner/runtime.py` has a force-push path based on producer silence. - - `showrunner/showrunner/autonomous_loop.py` waits only for low runway before calling that path. - - `stream_orchestrator/stream_orchestrator/supervisor.py` exits on external-producer `idle_timeout`. - - Real log evidence: the latest run accepted pushes at `22:10:44Z`, `22:11:07Z`, and `22:11:44Z`, then ended with `idle_timeout` at `22:12:05Z`. -- P1: Browser sync itself is not the main blocker. - - Marker logs show `A-player` cue firing with drift roughly `10ms`, `23ms`, `44ms`, and `48ms`. - - `script_producer/script_producer/word_tracker.py` already follows the latest player clock anchor instead of relying on stale fixed wall-clock sleeps. -- P1: The autonomous LLM planner is too one-shot. - - It can choose a URL and also author page-specific cues before the browser has inspected that URL. - - This should become a two-phase scout/observe/author/validate loop. -- P2: Cue schemas are duplicated and divergent. - - `showrunner/showrunner/cues.py` supports tab/history ops. - - `script_producer/script_producer/cues.py` only supports the original v1 browser ops. -- P3: `BrowserPipeline` waits again for positive `lead_ms`, even though `WordTracker` already applies lead timing. -- P3: Root docs are partially stale. Some documented open issues are fixed in code, and some paths refer to older layouts. - -### Current Recommendation - -- Keep the v1 demo as a constrained local Playwright "live research desk" with synced highlights, scrolls, and public-page navigation. -- Do not switch the core runtime to a generic browser-agent framework yet. -- Use browser-agent tools only as research/scout inspiration until streaming stability is solved. - -### Next Patch - -- Fix producer-silence wakeup in `showrunner/showrunner/autonomous_loop.py`. -- Make external-producer `idle_timeout` recoverable in `stream_orchestrator/stream_orchestrator/supervisor.py`. -- Add a bridge segment fallback for silence danger when no committed segment is ready. -- Add tests for high runway plus high producer silence. -- Then run a real 10-30 minute stream soak before changing the personality or demo concept further. - -### Implemented - -- Completed first Phase 0 stream-stability patch: - - `AutonomousLoopConfig` now has `producer_silence_force_push_sec`. - - `_wait_until_runway_low()` can wake on producer-silence danger when that fallback is enabled. - - Default showrunner behavior is now runway-only for long-idle/dedicated-queue sessions. - - External-producer `idle_timeout` now logs recovery intent instead of ending the supervisor loop. - - Added regression tests for high-runway/high-silence pusher wakeup and external idle recovery. -- Added a multi-phase roadmap to `PLAN.md` covering stream stability, browser reliability, personality, chat input, high-wow activity screens, and 24/7 operations. -- Clarified and hardened the existing real-browser direction: - - The codebase already had a real public-page path via `tools.run_autonomous_loop` and `BrowserExecutor`. - - Production activity screen should remain real headed Google Chrome on public pages. - - The dashboard/demo surface is smoke-test-only. - - `tools.run_autonomous_loop --fullscreen` now launches headed mode and resolves `--browser-channel auto` to Google Chrome. - - Added browser polish knobs for cursor speed and smooth-scroll cadence. - - Highlights now try native browser text selection before falling back to the overlay focus box. - - Cursor styling now maps common CSS cursor states to the visible overlay cursor. -- Added stream layout and autonomy controls: - - `tools.run_autonomous_loop` now has `--layout-preset` with `obs-1080p`, `obs-720p`, `laptop-16x9`, and `custom`. - - Default live layout is now `obs-1080p` (`1920x1080`) instead of the laptop/debug `1280x800` shape. - - Added `--autonomy-mode focused|curious|free` and `--show-goal`. - - Planner instructions now explicitly enforce viewport discipline: if viewers cannot see it, scroll before discussing/highlighting it. - -### Verified After Patch - -- Focused tests: - - `showrunner/tests/test_autonomous_loop.py` -> `13 passed` - - `stream_orchestrator/tests/test_supervisor_recovery.py` -> `3 passed` - - `showrunner/tests/test_runtime.py tests/test_autonomous_loop.py tests/test_browser_executor.py tests/test_bezier_path.py` -> `39 passed` - - `showrunner/tests/test_llm_scene_planner.py tests/test_vision_planner.py tests/test_runtime.py tests/test_autonomous_loop.py tests/test_browser_executor.py tests/test_bezier_path.py` -> `56 passed` - - `tools.run_autonomous_loop --help` verified the new layout/autonomy flags render correctly. -- Full tests: - - `showrunner`: `162 passed` - - `stream_orchestrator`: `123 passed` - - `script_producer`: `34 passed` -- Local smokes: - - `tools.smoke_full_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - `tools.smoke_autonomous_loop_fake` passed with `browser_actions=3`, `marker_events=3`. - - `tools.smoke_browser_executor` passed navigate, highlight, point, type, click, scroll, highlight. - - Re-ran all three after the layout/autonomy/browser-polish changes; all still passed. - -## 2026-05-24 — Real Showrunner E2E Trace - -Implemented `showrunner/tools/trace_live_e2e.py`, a black-box recorder for real M1 + showrunner runs. It watches `/debug/state`, `/producer/words`, `/producer/session_clock`, `/activity/segments`, and `/control/sessions`, then writes: - -- `timeline.jsonl` -- `summary.json` -- cue/browser lag distributions -- player-clock flat runs and backward-reset detection -- LLM latency by phase -- pushed-segment runway snapshots - -Real run: - -```bash -cd /Users//live-streamer/showrunner -uv run python -m tools.trace_live_e2e --duration-sec 420 --out-dir /tmp/showrunner_e2e_trace_20260524T103858Z --include-word-text -uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --max-scenes 5 --timeout-sec 420 \ - --event-log /tmp/showrunner_e2e_trace_20260524T103858Z/showrunner.sqlite \ - --marker-log /tmp/showrunner_e2e_trace_20260524T103858Z/markers.jsonl \ - --screenshot /tmp/showrunner_e2e_trace_20260524T103858Z/final_browser.png -``` - -Observed: - -- `5` scenes pushed and completed. -- `8` CUE events fired, all on `A-player`. -- Browser executed `7` actions and recorded `1` browser target error. -- Marker drift: mean `274ms`, min `143ms`, max `478ms`. -- Claude latency: scout mean `2.96s`; scene mean `4.84s`, max `8.55s`. -- Direct runway used `player_clock` for every push. Example proof: final old producer estimator was about `-53s` while player-clock runway was `21.73s`. -- Player clock stayed fresh, median anchor age `122ms`. -- Media/player flat runs were measurable; trace recorded `9` flat runs over `1s`, mean `9.61s`, max `20.55s`. -- A post-show teardown reset moved `video.currentTime` from about `123.79s` to `4.13s`. This happened after showrunner stopped, but it exposed a real guardrail need. - -Hardening added after the trace: - -- `SessionClock` now rejects large backward `video.currentTime` jumps from the current primary player instead of accepting a reset as fresh runway. -- Noisy per-anchor logs moved from `INFO` to `DEBUG`. -- Trace tool now handles `SIGINT`/`SIGTERM`, summarizes errors compactly, splits LLM latency by phase, reports player-clock anomalies, and auto-stops after sustained M1 `/debug/state` unreachable time. - -Follow-up issues found: - -- Planner still emitted one overlong segment: `20.80s` versus the `20s` hard cap. -- Browser cue target resolution failed once on an invisible GitHub text target: `text=clangd-lsp`. -- Fullscreen headed Chrome used the native Mac page viewport (`1700x1342` screenshot), not a fixed `1920x1080` OBS viewport. Need a layout pass. - -Verification: - -- `cd stream_orchestrator && uv run pytest tests/test_session_clock.py tests/test_direct_producer.py` -> `28 passed` -- `cd script_producer && uv run pytest tests/test_word_tracker_timing.py` -> `23 passed` -- `cd showrunner && uv run python -m py_compile tools/trace_live_e2e.py` -> passed -- Full suites after hardening: - - `stream_orchestrator`: `136 passed` - - `script_producer`: `46 passed` - - `showrunner`: `219 passed` - -## 2026-05-24 — Fixed OBS Browser Layout - -Fixed the headed Chrome layout mismatch found in the real E2E trace. - -Problem: - -- `tools.run_autonomous_loop --headed --fullscreen` previously used Chrome's native macOS viewport. -- The final E2E screenshot was `1700x1342`, not `1920x1080`. -- On Retina/native fullscreen probes, Chrome can report a `1496x776` CSS viewport and capture `2992x1552` device pixels. That is wrong for OBS and wrong for LLM screenshot consistency. - -Changes: - -- `tools.run_autonomous_loop` now uses a fixed OBS viewport by default, even in headed Chrome. -- `--native-viewport` was added as an explicit manual-debug escape hatch. -- `--emulated-viewport` remains accepted as a deprecated no-op alias because fixed viewport is now the default. -- Final run JSON now includes `browser.actual_viewport` and `browser.screenshot_size`. -- Added `tools.probe_browser_layout` to measure browser metrics and screenshot dimensions without starting HeyGen. -- Other headed browser tools now pass `use_native_viewport=False` by default: command lab, autonomous scene, scripted show, LLM eval, browser pipeline, scene matrix, and LLM smoke observe path. - -Measured probes: - -- Fixed headed OBS probe: - - command: `cd showrunner && uv run python -m tools.probe_browser_layout --headed --layout-preset obs-1080p --screenshot /tmp/showrunner_layout_headed_fixed.png` - - result: configured `1920x1080`, executor viewport `1920x1080`, screenshot `1920x1080`. -- Native fullscreen probe: - - command: `cd showrunner && uv run python -m tools.probe_browser_layout --headed --native-viewport --fullscreen --layout-preset obs-1080p --screenshot /tmp/showrunner_layout_headed_native_fullscreen.png` - - result: executor viewport `1496x776`, screenshot `2992x1552`. - -Verification: - -- `cd showrunner && uv run pytest tests/test_run_autonomous_loop_layout.py tests/test_browser_executor.py -q` -> `20 passed` -- `cd showrunner && uv run pytest -q` -> `222 passed` -- `cd showrunner && uv run python -m py_compile ...` for changed browser tools -> passed -- `cd showrunner && uv run python -m tools.run_autonomous_loop --help | rg 'native-viewport|emulated-viewport|layout-preset'` -> verified -- `git diff --check` -> passed diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..ec58566 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,65 @@ +# Contributing + +Thanks for your interest in `live-streamer`. This is a working demo project — the +bar for contributions is "keep it running and keep the sync tight." + +## Repository layout + +The repo is four independent Python packages, each with its own `uv`-managed +virtualenv and its own `pytest` suite: + +| Package | What it does | +|---|---| +| `stream_orchestrator/` | M1 — HeyGen Avatar Realtime session lifecycle + HTTP/SSE contract + OBS browser sources | +| `showrunner/` | The content brain — LLM scene planner, Playwright browser executor, cue scheduler | +| `chat_ingester/` | Twitch chat reader → URL filter → SQLite queue the showrunner consumes | +| `script_producer/` | Minimal library slice imported by the showrunner (`word_tracker`, `markers`, `cues`, …) | + +## Prerequisites + +- **Python 3.11** (every package pins `requires-python = ">=3.11"`) +- **[uv](https://github.com/astral-sh/uv)** for dependency and venv management +- **Playwright + Chromium** (for the showrunner): `uv run playwright install chromium` +- **OBS Studio** with the obs-websocket plugin (for the streaming/egress path) +- A **HeyGen** API key (Avatar Realtime) and an **Anthropic** API key + +## Setup + +```bash +# 1. Copy the env template and fill in your keys +cp .env.example .env && $EDITOR .env # HEYGEN_API_KEY + ANTHROPIC_API_KEY at minimum + +# 2. Create each package's venv (run inside each package directory) +cd stream_orchestrator && uv sync && cd .. +cd showrunner && uv sync && uv run playwright install chromium && cd .. +cd chat_ingester && uv sync && cd .. +cd script_producer && uv sync && cd .. +``` + +## Running the tests + +Each package has its own suite. Run from inside the package directory: + +```bash +cd stream_orchestrator && uv run pytest -q +cd showrunner && uv run pytest -q +cd chat_ingester && uv run pytest -q +cd script_producer && uv run pytest -q +``` + +The `showrunner/` suite carries opt-in live tests (real Anthropic + real Chromium) +behind the `live` marker; they are deselected by default. Run them explicitly with: + +```bash +cd showrunner && uv run pytest -m live -q # requires ANTHROPIC_API_KEY +``` + +## Pull request expectations + +- Keep changes scoped and describe what you changed and why. +- Run the affected package's `pytest` suite before opening a PR; add or update tests + for behavior you change. +- The sync/pacing logic (word-timestamp scheduling, runway/buffer policy) is + load-bearing and easy to break subtly — call out any change that touches it, and + prefer a live smoke run when you do. +- Don't commit a real `.env` or any API keys. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..95706d1 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 HeyGen + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 888b225..005f673 100644 --- a/README.md +++ b/README.md @@ -1,139 +1,158 @@ # live-streamer -24/7 Twitch live-streamer: a HeyGen photo avatar narrates a live browser show. Playwright drives a real visible Chrome window through navigations, scrolls, highlights, and clicks. HeyGen Avatar Realtime v3 generates the avatar's speech with word-level timestamps so the cursor moves when the avatar says "let me click here" — not five seconds later. - -This is a standalone extraction from a broader internal HeyGen repo. Everything needed to run is in this directory. +**A 24/7 AI live-streamer.** A HeyGen photo avatar hosts a continuous Twitch show: +it drives a real, visible Chrome window — navigating, scrolling, highlighting, and +clicking through live web pages — and narrates what it sees in real time. Viewers +drop GitHub repositories in chat; the avatar pulls them up and roasts them on air. + +The load-bearing trick is **word-level timestamp sync**. HeyGen's Avatar Realtime +API returns per-word timestamps for the avatar's speech, so an on-screen cursor +moves at the exact moment the avatar says "let me click here" — not five seconds +before or after. Speech, video, and browser action stay locked to one clock. + +## How it works + +Four independent Python packages cooperate. The showrunner is the brain; the +stream orchestrator (M1) owns the HeyGen session and the OBS-facing surfaces. + +```mermaid +flowchart LR + viewer([Twitch viewers]) -->|drop GitHub URLs| chat[Twitch chat] + chat --> ingester[chat_ingester
IRC reader + URL filter
Haiku 4.5 pre-bakes acks] + ingester -->|SQLite queue| showrunner + + subgraph showrunner["showrunner (the content brain)"] + author[Episode author
Claude Opus 4.8] + reactive[Reactive planner
Claude Sonnet 4.6] + executor[Browser executor
Playwright] + tracker[word_tracker
cue scheduler] + end + + showrunner -->|push scene:
narration + browser plan + cues| m1[stream_orchestrator M1
session lifecycle + HTTP/SSE] + m1 <-->|text in / audio + HLS + per-word SSE| heygen[(HeyGen
Avatar Realtime API)] + showrunner -->|navigate / scroll / highlight / click| chrome[Visible Chrome window] + + m1 -->|avatar video + captions| obs[OBS Studio] + chrome -->|window capture| obs + obs -->|RTMP| twitch([Twitch]) +``` -## Quick start +**The roast lifecycle**, from a URL in chat to a verdict on screen: -Use the scripts when you do not want to remember flags. +```mermaid +flowchart TD + A[Viewer drops a GitHub URL in chat] --> B[chat_ingester filters + queues it
Haiku pre-bakes an acknowledgement] + B --> C[Episode author Opus 4.8 fetches the repo
and writes a scene: narration + browser plan + cues] + C --> D[Browser executor drives Chrome to the repo
via Playwright] + D --> E[M1 sends narration to HeyGen;
per-word timestamps stream back] + E --> F[word_tracker fires each cue on the spoken word
cursor, highlight, and scroll land in sync] + F --> G[Verdict is stored and shown on the leaderboard overlay] +``` -```bash -# One-time setup -cp .env.example .env && $EDITOR .env # fill ANTHROPIC_API_KEY + Twitch creds -scripts/01-login-browser-profile.sh # one-time Google login in the show's Chrome profile - # (stop with Ctrl-C before the autonomous run) +## Prerequisites -# Terminal 1 — M1 (HeyGen session manager + OBS browser sources, port 8765) -scripts/02-start-avatar.sh +- **Python 3.11** (every package pins `requires-python = ">=3.11"`) +- **[uv](https://github.com/astral-sh/uv)** for dependency / virtualenv management +- **OBS Studio** with the obs-websocket plugin (streaming + egress heartbeat) +- **Chrome / Chromium** (the showrunner installs its own via Playwright) +- A **HeyGen** API key (Avatar Realtime) and an **Anthropic** API key +- **macOS** if you want the unattended `launchd` operation described in + [docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md); the core + packages themselves are OS-agnostic -# Terminal 2 — autonomous local headed Chrome showrunner -source .env && scripts/03-run-autonomous-show.sh +## Setup -# Terminal 3 (optional) — Twitch chat ingester (Module 4). -# Viewers drop GitHub URLs in chat, the avatar acknowledges them by name, -# navigates to the repo, and roasts it. REQUIRES `SHOWRUNNER_EPISODE_MODE=true` -# in the showrunner's env (already set if you sourced `.env`). -source .env && scripts/05-run-chat-ingester.sh +```bash +# Fill in your keys — one root .env serves all three runtime components. +cp .env.example .env && $EDITOR .env # ANTHROPIC_API_KEY + HEYGEN_API_KEY (+ Twitch creds) -# Optional — open local viewer/debug pages. -scripts/04-open-local-views.sh +# Create each package's uv virtualenv. +cd stream_orchestrator && uv sync && cd .. +cd showrunner && uv sync && uv run playwright install chromium && cd .. +cd chat_ingester && uv sync && cd .. +cd script_producer && uv sync && cd .. -# Cleanup if a local process is stuck. -scripts/00-stop-local-processes.sh +# One-time Google login in the show's dedicated Chrome profile (stop with Ctrl-C). +scripts/01-login-browser-profile.sh ``` -**For unattended 24/7 operation on a Mac Mini**, see [docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md) — covers launchd plists, system settings, OBS heartbeat, and a 7-day soak checklist. +The login script and the autonomous show share the same dedicated Chrome profile +(`.browser-profile` by default). Don't run them at the same time — Chrome profiles +are single-writer. Use a dedicated Google account, not your personal one. -The login script and autonomous script share the same dedicated Chrome profile at `.browser-profile` by default. Do not run the login lab and autonomous show at the same time; Chrome profiles are single-writer state. Use a dedicated Google account, not your personal/default Chrome profile. +## Running the show -Manual equivalent, three terminals. M1 first; showrunner brings up its own browser. +The `scripts/` wrappers exist so you don't have to remember flags. Bring up M1 +first; the showrunner spawns its own browser. ```bash -# Terminal 1 — Module 1 (HeyGen session manager + OBS browser sources) -cd stream_orchestrator -uv sync -export HEYGEN_API_KEY=... # or put it in a repo-root .env (see .env.example) -STREAM_ORCHESTRATOR_EXTERNAL_PRODUCER_MODE=1 uv run python -m stream_orchestrator -# Wait for `supervisor.session_ready` in logs. - -# Terminal 2 — showrunner (LLM planner + Playwright browser) -cd showrunner -uv sync -uv run playwright install chromium # one-time -export ANTHROPIC_API_KEY=... -uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --max-scenes 3 --timeout-sec 600 - -# (Optional) open the eye-check debug UI in a regular browser tab -open http://127.0.0.1:8765/debug -``` +# Terminal 1 — M1 (HeyGen session manager + OBS browser sources, port 8765) +scripts/02-start-avatar.sh +# wait for `supervisor.session_ready` in the logs -**OBS scene**: -- Browser source at `http://127.0.0.1:8765/` (the avatar, 1920×1080) -- Browser source at `http://127.0.0.1:8765/captions` (1920×220, position over avatar) -- Window Capture of the Playwright Chrome window the showrunner spawns (the main scene) +# Terminal 2 — the autonomous, headed-Chrome showrunner +source .env && scripts/03-run-autonomous-show.sh -## What lives where +# Terminal 3 (optional) — Twitch chat ingester (Module 4) +# Viewers drop GitHub URLs; the avatar acknowledges them by name, navigates, +# and roasts. Requires SHOWRUNNER_EPISODE_MODE=true (already set if you sourced .env). +source .env && scripts/05-run-chat-ingester.sh -| Directory | Role | Runs as a process | -|---|---|---| -| [`stream_orchestrator/`](stream_orchestrator/) | M1 — HeyGen session lifecycle, hot-swap HLS rotation, the integration surface for everything else. Owns the OBS browser sources (`/`, `/captions`, `/activity`, `/debug`) and the `/producer/*` + `/activity/segments` HTTP contract. | ✅ yes | -| [`showrunner/`](showrunner/) | The content brain. LLM planner authors scenes (narration + browser action plan + cues); browser_executor performs them via Playwright; word_tracker fires cues on HeyGen's per-word timestamps so visuals land in sync with speech. | ✅ yes | -| [`script_producer/`](script_producer/) | Legacy minimal slice. Only 5 files (`word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`) — the showrunner imports these as a library via a sys.path hack in `showrunner/tools/*.py`. **No M2 daemon process runs.** | ❌ never started; library import only | +# Optional helpers +scripts/04-open-local-views.sh # open local viewer / debug pages +scripts/00-stop-local-processes.sh # clean up a stuck local process +``` -## Docs +**OBS scene** — three sources pointed at the running M1: -- **[TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md)** — start here. Architecture diagram, sync model (three clocks + the anchor that bridges them), end-to-end cue trace, current open problems P0-P5, file map, asks for help. Written for teammates jumping in. -- **[STATUS.md](STATUS.md)** — running engineering changelog. What changed when and why. -- **[showrunner/DEMO_RUNBOOK.md](showrunner/DEMO_RUNBOOK.md)** — showrunner-specific demo commands and SQLite event-log queries. -- **[stream_orchestrator/RUN.md](stream_orchestrator/RUN.md)** — M1 operational notes. -- **[stream_orchestrator/ARCHITECTURE.md](stream_orchestrator/ARCHITECTURE.md)** — M1 design doc. -- **[research/CLOUD_BROWSER_STRATEGY.md](research/CLOUD_BROWSER_STRATEGY.md)** — Bright Data / Browser Use Cloud review and how they should fit without breaking local media-clock sync. +- Browser source `http://127.0.0.1:8765/` — the avatar (1920×1080) +- Browser source `http://127.0.0.1:8765/captions` — captions strip (1920×220, over the avatar) +- Window Capture of the Playwright Chrome window the showrunner spawns (the main view) -## Test suites +For unattended 24/7 operation on a Mac Mini (launchd plists, system settings, OBS +heartbeat, a 7-day soak checklist), see +[docs/operations/mac-mini-setup.md](docs/operations/mac-mini-setup.md). -Each package has its own venv (managed by `uv`). Current counts as of 2026-06-04: +## Tests -```bash -cd stream_orchestrator && uv run pytest -q # 213 tests -cd ../showrunner && uv run pytest -q # 1362 tests (+7 deselected live tests) -cd ../chat_ingester && uv run pytest -q # 114 tests -cd ../script_producer && uv run pytest -q # 53 tests (subset of legacy suite) -# Total: 1,742 tests, all green -``` +Each package ships its own `pytest` suite. Run from inside the package directory: -Opt-in live tests (real Anthropic + real Chromium) run separately: ```bash -cd showrunner && uv run pytest -m live -q # 7 tests; requires ANTHROPIC_API_KEY +cd stream_orchestrator && uv run pytest -q +cd showrunner && uv run pytest -q +cd chat_ingester && uv run pytest -q +cd script_producer && uv run pytest -q ``` -## Why the legacy `script_producer/` directory still exists - -The showrunner needs `word_tracker.py` (~750 lines of marker-parsing + word-timestamp scheduling). That file lived in `script_producer/script_producer/` in the original repo. Rather than refactor it into the showrunner package, we kept it where it was and the showrunner imports it via: - -```python -# from showrunner/tools/run_autonomous_loop.py -_REPO_ROOT = Path(__file__).resolve().parents[2] -for _path in (_REPO_ROOT, _REPO_ROOT / "showrunner", _REPO_ROOT / "script_producer"): - if str(_path) not in sys.path: - sys.path.insert(0, str(_path)) +The `showrunner/` suite additionally carries opt-in live tests (real Anthropic + +real Chromium) behind the `live` marker, deselected by default: -from script_producer.word_tracker import WordTracker +```bash +cd showrunner && uv run pytest -m live -q # requires ANTHROPIC_API_KEY ``` -This standalone repo keeps that structure so the import works without changes. The only files in `script_producer/script_producer/` are the 5 that the showrunner actually imports. Everything else (the M2 daemon's tier_generator, Claude client, GitHub fetcher, etc.) was dropped. - -**Cleanup that's NOT yet done**: properly fold `word_tracker.py` + `markers.py` into the `showrunner/` package, drop the sys.path hack, delete `script_producer/`. Mechanical work, ~1-2 hours. See `script_producer/script_producer/LEGACY_REPO_REVIEW_PROTOTYPE.md` in the original repo for the intent. - -## What was excluded from the extraction - -- **`activity_screen/`** (Module 3) — Playwright PNG screenshotter for the old `/activity` browser source. Unused in the showrunner architecture. If you need it back, it lives in the original internal repo under `activity_screen/`. -- **HeyGen's backend service** — the Avatar Realtime v3 server source lives there; consult HeyGen for server-behavior debugging. -- **`ai-twitch-streamer/`** — a prior abandoned attempt at this same idea. Explicitly excluded. - -## Current state and known problems - -Episode v2 (one-LLM-call cinematic roasts) and §7+§8 (autonomous trending discovery + episode transitions + chat callbacks + link curiosity) are in. CostCap, Notifier, VerdictStore + leaderboard overlay, cross-session anti-repeat (Plan 8) are wired. Twitch chat ingester (Module 4) is built and waiting on env vars to engage. See [docs/superpowers/specs/](docs/superpowers/specs/) for the design history. - -Open work, in priority order: - -| Item | Status | -|---|---| -| Plan 9 — chat-pulse + @-mentions (LLM reacts to non-URL chat) | Spec written ([docs/superpowers/specs/2026-05-28-chat-pulse-and-mentions-design.md](docs/superpowers/specs/2026-05-28-chat-pulse-and-mentions-design.md)); not implemented | -| `word_tracker.prefix_timeout` | Legacy M2 code path; surfaced under load. Out of scope for showrunner-side fixes | -| OBS heartbeat → showrunner.notify consolidation | Documented in `.env.example` (point both webhook envs at same URL); real unification deferred | -| Long-soak live verification | Needs ≥1 hour live run on the Mac Mini setup | - -Tests pass (1,197 across all four suites). The known operational gotcha: chat ingestion requires `SHOWRUNNER_EPISODE_MODE=true` AND `CHAT_QUEUE_DB=state/chat_queue.sqlite` in the showrunner's env — without these, chat rows are written but never read. The `.env.example` template documents this contract. - +## Architecture +| Directory | Role | Long-running process? | +|---|---|---| +| [`stream_orchestrator/`](stream_orchestrator/) | **M1** — HeyGen Avatar Realtime session lifecycle, hot-swap HLS rotation, and the OBS browser sources (`/`, `/captions`, `/debug`) plus the `/producer/*` and `/activity/segments` HTTP/SSE contract. | yes | +| [`showrunner/`](showrunner/) | The content brain. The LLM scene planner authors scenes (narration + browser action plan + cues); the browser executor performs them via Playwright; `word_tracker` fires cues on HeyGen's per-word timestamps so visuals land in sync with speech. | yes | +| [`chat_ingester/`](chat_ingester/) | Twitch chat reader → URL filter → SQLite queue the showrunner consumes. Pre-bakes per-message acknowledgements with Claude Haiku 4.5. | yes | +| [`script_producer/`](script_producer/) | Minimal library slice imported by the showrunner (`word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`). No standalone daemon runs. | no (import only) | + +Start with **[TECHNICAL_OVERVIEW.md](TECHNICAL_OVERVIEW.md)** — the architecture +diagram, the three-clocks sync model and the anchor that bridges them, an +end-to-end cue trace, and the current open problems. Other docs: + +- [stream_orchestrator/ARCHITECTURE.md](stream_orchestrator/ARCHITECTURE.md) — M1 design doc +- [stream_orchestrator/RUN.md](stream_orchestrator/RUN.md) — M1 operational notes +- [showrunner/DEMO_RUNBOOK.md](showrunner/DEMO_RUNBOOK.md) — showrunner demo commands and event-log queries +- [chat_ingester/README.md](chat_ingester/README.md) — chat pipeline details +- [HOST_CHARACTER.md](HOST_CHARACTER.md) — the host's persona and voice rules +- [docs/superpowers/specs/](docs/superpowers/specs/) — feature design specs +- [research/](research/) — browser-agent and cloud-browser research notes + +## License + +Released under the [MIT License](LICENSE). See [CONTRIBUTING.md](CONTRIBUTING.md) +to get set up for development. diff --git a/REPORT.md b/REPORT.md deleted file mode 100644 index 0300c48..0000000 --- a/REPORT.md +++ /dev/null @@ -1,224 +0,0 @@ -# live-streamer — reliability + cleanup pass - -**Date:** 2026-07-10 · **Branch:** `fix/reliability-and-cleanup` - -This is a senior-engineering pass over the repo with three goals: **fix the real -reliability bugs**, **simplify aggressively**, and **make the source safe to open -to the public**. Every claim below was verified at source and, where possible, -proven with a test or a live API call — not taken from the repo's own prose -(which was treated as unreliable). - -TL;DR of what changed: -- **3 reliability bugs fixed** (words-stale over-push, orphaned remote sessions, - event-loop-blocking I/O) — each with regression tests. Orchestrator suite: - **213 → 217 passing**. Showrunner suite: **1362 passing**. -- **~4,000 LOC removed** (19 dead dev/probe/smoke harnesses) + two misleading - code artifacts corrected. -- **Public-safety scrub**: 0 live secrets in the tree, personal username/paths, - internal host, avatar/voice IDs, and an internal ops runbook all removed. -- **Live-validated** the HeyGen v3 control-plane contract (create/poll/cancel) - against both `api.heygen.com` and the canary — spending real credits. - ---- - -## 1. Reliability fixes - -### P0-1 — Words-stale fuse drove an unbounded over-push (FIXED) -`stream_orchestrator/http_server.py` · `_direct_runway_snapshot` - -**Bug:** the pusher gate trusts `runway_sec` = backlog `B = latest_word_end − -currentTime`. When HeyGen's per-word SSE goes quiet for >25s (its *normal* bursty -pattern), the code fell back to `fused = min(estimated, _FUSE_CLAMP_SEC)`. The -wall-clock `estimated` **drifts negative on long sessions**, so it leaked through -`min()` → gate read "starving" → pusher force-fed → the TTS backlog ballooned to -~90s and cues desynced from speech. - -**Fix:** on the stale path, when a word has ever arrived, use the *draining* real -backlog `B = word_end − currentTime` clamped to `[0, _FUSE_CLAMP_SEC]` (never -negative, drains correctly as the avatar plays through the buffer). Only in true -warmup (no word ever seen) do we fall back to the clamped wall estimate. - -This is the fix that was authored on the abandoned `fix/words-stale-fuse-overpush` -branch but **never merged** — I ported only the `http_server.py` hunk onto `main` -(dropping the branch's unrelated cruft: a hardcoded voice_id and a -`.python-version`). *Notably, that branch never updated the unit test that pins -this behavior — which is almost certainly why it failed CI and was abandoned.* - -**Verified:** the pre-existing test that asserted the buggy behavior was rewritten -to encode the correct behavior, plus two new regression tests covering **both** -failure directions the old `min(estimate, clamp)` caused: -- `test_runway_stale_fuse_ignores_negative_wall_estimate` — negative drift no - longer signals false starvation (the balloon). -- `test_runway_stale_fuse_drains_to_zero` — a *high* stale estimate no longer - reports a false-full buffer (which would starve the avatar → dead air). - -### P0-3 — Recovery orphaned the remote HeyGen session (FIXED) -`stream_orchestrator/supervisor.py` · `_teardown_current` - -**Bug:** teardown only sent HeyGen `/cancel` for three *graceful* end reasons -(`supervisor_exit`, `supervisor_cancelled`, `bootstrap_timeout`). Every session -**death** reason from `_classify_death` (`feeder_transient_exhausted`, -`poller_api_error`, `feeder_api_error`, `unknown`, …) reached teardown via the -recovery path and **fell straight through** — the local process abandoned the -`stream_id` without telling HeyGen. The orphaned workflow kept burning -GPU/inference on the shared, single-session account until its idle timeout, and -(since the account is single-session) contended with the replacement session. - -**Fix:** release the remote session on **any** teardown where it might still be -alive. Skip only when it's already known-gone (`session.finalized`, the 404 -`remote_gone` latch, or a terminal `last_status`). On a death we go straight to -`/cancel` (fast slot release, no wasted final-flush grace window); graceful stops -of a healthy streaming session keep the final-first behavior. `/cancel` is -idempotent, so an extra cancel on a dying session is a harmless no-op. - -**Verified:** two new tests — `test_teardown_on_session_death_cancels_remote_not_orphaned` -(death → `/cancel`, no orphan, no flush) and `test_teardown_skips_cancel_when_remote_already_gone` -(the inverse guard). All 8 pre-existing recovery tests still pass. - -### P0-4 — Blocking I/O froze the single shared event loop (FIXED) -`stream_orchestrator/notify.py`, `http_server.py` - -The orchestrator runs uvicorn + supervisor on **one** event loop, so any -synchronous blocking call freezes the feeder, poller, and all SSE streams at once. - -- **`notify_operator`** did a synchronous `urllib` POST (up to 2s) and was called - un-awaited from the supervisor's async recovery loop — freezing the stream for - up to 2s *exactly during the incident that triggered the alert*. Added - `notify_operator_async` (offloads to a worker thread via `asyncio.to_thread`); - the three call sites now `await` it. -- **Three OBS-overlay handlers** did synchronous `sqlite3.connect`+query - (`_read_chat_queue_overlay`, `_read_leaderboard_overlay`, `_read_show_stats`) - directly on the loop — every overlay poll could stall it up to the 0.5s lock - timeout. All three now run via `asyncio.to_thread`. - -**Verified:** full orchestrator suite (217) green, including the Discord-alert and -overlay tests. - -### P0-2 — Auto-recovery is disabled (NO CHANGE — deliberate, documented) -`stream_orchestrator/__main__.py` (starvation watchdog commented out) - -The initial investigation flagged "zero automated self-healing." On closer -reading **that framing is overstated**: genuine session death *is* auto-recovered -— the poller rotates on terminal status and `remote_gone` rotates on a 404. What -the disabled watchdog covered was a *non-terminal silent freeze*, and the operator -disabled it 2026-06-04 with live evidence: its 12s kill was converting brief, -self-healing feed-gap freezes into ~46s cold rotations (worse than the stall). - -Re-enabling it blindly would reintroduce exactly that flapping and make the system -*less* reliable — so I did **not** change runtime behavior here. The correct way to -close the remaining gap (a sustained, never-thawing freeze) is a **hysteresis -watchdog** with a much longer threshold (e.g. rotate only after a >60–90s freeze, -ignore brief blips). The `StarvationWatchdog` code is retained and wired for a -one-line re-enable. **This needs a live soak to tune the threshold safely** — see -§5. - -### P1-10 — Ambiguous 4xx treated as death (MITIGATED by P0-3) -A transient 4xx from the feeder becomes a generic `feeder_api_error` → session -death → full recreate. With P0-3 fixed, this no longer *orphans* anything (the -remote is now always released). Retrying transient 4xx **in-session** instead of -tearing down is a further optimization that needs a live soak — deferred to §5. - ---- - -## 2. Simplification - -- **Removed 19 dead dev/probe/smoke harness scripts** (~4,000 LOC) from - `showrunner/tools/` and `stream_orchestrator/tools/` — none imported by any - production module or invoked by `scripts/*.sh`. The real `tests/` suites are - untouched. Kept: `run_autonomous_loop.py` (prod entry), `run_autonomous_scene.py` - (exports `FakeSceneClient`, used by prod entry), `browser_command_lab.py` (used - by the login flow), and `analyze_run.py`/`measure_media_playback.py` (have - tests / library dependents). Git history preserves everything removed. -- **`_effective_runway`** (`showrunner/autonomous_loop.py`): trimmed a 44-line - docstring describing a *retired* `min()` combiner down to an accurate 10-line - one. The implementation is 3 lines and just trusts `runway_sec` now. -- **`_pending_over_ceiling`**: the `getattr(..., 35.0)` fallback used the **wrong** - default (the real dataclass default is `25.0`) — a latent 10s-too-high ceiling - for any partial config. Corrected to `25.0`. - ---- - -## 3. Public-safety (open-source readiness) - -A full security audit found **no live secrets in the working tree or git -history** — every credential is env-loaded, `.env` was never committed, and -`.gitignore` coverage (state DBs, logs, chrome profiles, `.env`) is solid. - -Scrubbed for release: -- **Personal PII**: the operator's macOS username and `~/Desktop/...` paths across - all launchd plists and docs → placeholders (0 remaining hits). -- **Internal endpoint**: `api-canary.heygen.com` → the public `api.heygen.com` - (config default + docs). -- **Production resource IDs**: hardcoded `avatar_id`/`voice_id` → `` - placeholders. -- **Internal ops runbook** `docs/datadog-guide.md` (exposed the prod Datadog org, - an internal API-account id, and internal service names) → **deleted**. -- **Internal monorepo paths / Nacos keys / PR numbers / prod run IDs** → - genericized in docs and code comments. - -**Two items require the repo owner (I did not do these autonomously):** -1. **Rotate the GitHub PAT** embedded in the local clone's `.git/config` remote - URL. It is *not* published (GitHub never serves `.git/config`) and is not in - the tree or history, but it's a live credential in the workspace — assume - compromised and rotate. Then `git remote set-url origin` without the token. -2. **Optional history rewrite** for a few stale `.playwright-mcp/` DOM snapshots - (public docs pages, *no* secrets) if you want a pristine history. - -Residual low-sensitivity items flagged but left (they risk mangling technical -docs): ~40 descriptive "canary" prose mentions in internal design docs, and a few -8-hex truncated run IDs in kill-test tables. - ---- - -## 4. Live validation (real credits spent) - -I exercised the v3 realtime control plane live against **both** `api.heygen.com` -and `api-canary.heygen.com` using the account's API key (~396k credits available): -- `POST /v3/avatar-realtime` (create) → **201**, returns `stream_id`. ✅ -- `GET /v3/avatar-realtime/{id}` (poll) → **200**, status transitions. ✅ -- `POST /v3/avatar-realtime/{id}/cancel` → **202**, `cancelled: true`. ✅ (this is - exactly the release path P0-3 fixes.) - -**Finding:** on both endpoints the session **stayed `pending` for the full 90s and -never provisioned HLS to `streaming`** with the show's avatar/voice. Create and -cancel work; end-to-end provisioning requires account/space-side realtime -enablement (or a realtime-provisioned avatar) that this API key doesn't have from -this environment. So I could not capture a full live words-SSE trace here — the -fuse fix is instead proven by the unit tests, a numerical model, and the original -commit's own "validated live" note. (The probe script is at -`live-streamer-investigation/live_v3_probe.py` if you want to re-run it with a -realtime-enabled avatar.) - ---- - -## 5. Recommended follow-ups (need a live soak, not a code-review change) - -These are real improvements but changing them blind — without hours of live -Datadog observation — risks the exact freeze/overshoot oscillation this subsystem -has a long history of. Flagged, not done: -1. **Hysteresis starvation watchdog** (P0-2): re-enable with a long, debounced - threshold so a sustained freeze rotates but brief self-healing stalls don't. -2. **In-session retry of transient 4xx** (P1-10): distinguish transient vs - terminal API errors instead of tearing the session down on any 4xx. -3. **Collapse the three overlapping backlog caps** (`max_in_flight_scenes`, - `pending_speech_ceiling_sec`, the runway gate) toward the "1 signal + 1 fuse" - design the repo's own spec calls for — timing-sensitive, needs a soak. -4. **De-duplicate `cues.py`** (showrunner vs script_producer copies) and the - speech-rate `2.2` constant, and drop the `sys.path` import hack — cross-package - refactors; mechanically safe but want an integration run to confirm. - ---- - -## How to run the tests - -```bash -# orchestrator -cd stream_orchestrator && python -m venv .venv && . .venv/bin/activate \ - && pip install -e . pytest pytest-asyncio && python -m pytest -q # 217 passing - -# showrunner -cd showrunner && python -m venv .venv && . .venv/bin/activate \ - && pip install -e . pytest pytest-asyncio respx && python -m pytest -q # 1362 passing -``` - -— Reliability + cleanup pass by Ralu 🦦 diff --git a/TECHNICAL_OVERVIEW.md b/TECHNICAL_OVERVIEW.md index 6baad1f..1f39ea1 100644 --- a/TECHNICAL_OVERVIEW.md +++ b/TECHNICAL_OVERVIEW.md @@ -2,7 +2,7 @@ 24/7 HeyGen avatar narrating a live Playwright browser show. Word-level timestamp sync via HeyGen's `/v3/avatar-realtime/{id}/words` SSE. -Standalone extraction from a HeyGen monorepo. Everything runtime-required is in this directory. +A standalone project. Everything runtime-required is in this directory. --- @@ -27,13 +27,7 @@ uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --ma open http://127.0.0.1:8765/debug ``` -**Test suites** (run in each subdir): `uv run pytest -q` -- `stream_orchestrator/` → 167 tests -- `showrunner/` → 896 tests (+7 opt-in `@pytest.mark.live` deselected) -- `chat_ingester/` → 87 tests -- `script_producer/` → 47 tests (subset of original) - -**Total: 1,197 tests as of 2026-05-28.** +**Test suites**: each package ships its own `pytest` suite. Run `uv run pytest -q` inside each subdir (`stream_orchestrator/`, `showrunner/`, `chat_ingester/`, `script_producer/`). The `showrunner/` suite additionally carries 7 opt-in `@pytest.mark.live` tests (deselected by default; require real API keys). **Sanity checks** while running: ```bash @@ -54,7 +48,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \ ``` ┌─────────────────────────────────────────┐ - │ HeyGen Canary API │ + │ HeyGen Avatar Realtime API │ │ (text in → audio + HLS video out) │ └─────┬───────────────────────────┬───────┘ ▲ │ @@ -101,7 +95,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \ | [`showrunner/`](showrunner/) | LLM scene planner → browser_executor (Playwright) + word_tracker (cue scheduler). | yes | | [`script_producer/`](script_producer/) | Library import only. 5 files used by showrunner: `word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`. Full M2 daemon was dropped. | no | -Dropped from the extraction: `activity_screen/` (M3 PNG screenshotter, unused), full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), HeyGen's backend service source. +Not included: `activity_screen/` (M3 PNG screenshotter, unused), the full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), and HeyGen's backend service source. --- @@ -132,7 +126,7 @@ At `fire_wall_ms`, POST `/producer/marker_fired` → broadcast via `/activity/se **Why this works**: both `word.start` and `currentTime` are on the same axis. Anchor = the map to wall clock. Stalls freeze `currentTime`; re-anchor 250 ms later shifts `wall_t0` forward by the stall amount. -**Pipeline latency** (canary, measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s). +**Pipeline latency** (measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s). --- @@ -162,7 +156,7 @@ LLM authors `[CUE: {"op":"navigate","target":"https://github.com/foo/bar","lead_ | 1 | Emitter's `word.start` is accurate | ✅ HeyGen's responsibility, stable | | 2 | `