heygen-com · ukimsanov · Jul 12, 2026 · Jul 12, 2026
diff --git a/.env.example b/.env.example
@@ -28,7 +28,7 @@ HEYGEN_API_KEY=
 #   - chat_ingester (Haiku ack writer pre-bakes per-row ack text)
 # Same key for both — the showrunner uses Sonnet 4.6, chat_ingester
 # uses Haiku 4.5; both consume from the same account.
-ANTHROPIC_API_KEY=sk-ant-...
+ANTHROPIC_API_KEY=<anthropic-api-key>
 
 
 # ─── Optional: GitHub (richer repo fetch) ────────────────────────────
@@ -54,7 +54,7 @@ OBS_HEARTBEAT_PASSWORD=
 #
 # Tokens expire ~60 days; regenerate when you see
 # `twitch_reader.auth_failed` in chat_ingester logs.
-TWITCH_BOT_TOKEN=oauth:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TWITCH_BOT_TOKEN=oauth:<token>
 
 # The bot account's Twitch login (lowercase). Create a second Twitch
 # account or reuse one you own — this is who the token belongs to.
@@ -141,17 +141,17 @@ CHAT_QUEUE_DB=state/chat_queue.sqlite
 # If no sink is configured, alerts just log at WARNING.
 
 # Slack-compatible incoming webhook URL.
-# SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
+# SLACK_WEBHOOK_URL=<slack-webhook-url>
 
 # Discord-compatible incoming webhook URL.
-# DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
+# DISCORD_WEBHOOK_URL=<discord-webhook-url>
 
 # OBS heartbeat alerts (separate stdlib-only script under scripts/).
 # Point this at the SAME Slack/Discord webhook as above to consolidate
 # all alerts into one channel. The OBS heartbeat doesn't share the
 # Notifier rate-limit — set this to the same URL and you'll see OBS
 # alerts alongside CostCap + failure_alarm alerts.
-# OBS_HEARTBEAT_ALERT_WEBHOOK=https://hooks.slack.com/services/...
+# OBS_HEARTBEAT_ALERT_WEBHOOK=<slack-webhook-url>
 
 # macOS-native notification (osascript). '1' / 'true' / 'yes' enables;
 # anything else (or unset) disables. Useful for local development;

diff --git a/.gitignore b/.gitignore
@@ -43,6 +43,10 @@ Thumbs.db
 *.swp
 *~
 
+# Assistant-local workspace metadata
+.claude/
+.codex/
+
 .browser-profile/
 
-logs/
+logs/
diff --git a/.mcp.json b/.mcp.json
@@ -0,0 +1,8 @@
+{
+  "mcpServers": {
+    "temporal-docs": {
+      "type": "http",
+      "url": "https://temporal.mcp.kapa.ai"
+    }
+  }
+}
diff --git a/CHANGELOG.md b/CHANGELOG.md
diff --git a/README.md b/README.md
diff --git a/TECHNICAL_OVERVIEW.md b/TECHNICAL_OVERVIEW.md
@@ -2,7 +2,7 @@
 
 24/7 HeyGen avatar narrating a live Playwright browser show. Word-level timestamp sync via HeyGen's `/v3/avatar-realtime/{id}/words` SSE.
 
-A standalone project. Everything runtime-required is in this directory.
+Standalone extraction from a HeyGen monorepo. Everything runtime-required is in this directory.
 
 ---
 
@@ -27,7 +27,13 @@ uv run python -m tools.run_autonomous_loop --real-llm --headed --fullscreen --ma
 open http://127.0.0.1:8765/debug
 ```
 
-**Test suites**: each package ships its own `pytest` suite. Run `uv run pytest -q` inside each subdir (`stream_orchestrator/`, `showrunner/`, `chat_ingester/`, `script_producer/`). The `showrunner/` suite additionally carries 7 opt-in `@pytest.mark.live` tests (deselected by default; require real API keys).
+**Test suites** (run in each subdir): `uv run pytest -q`
+- `stream_orchestrator/` → 167 tests
+- `showrunner/` → 896 tests (+7 opt-in `@pytest.mark.live` deselected)
+- `chat_ingester/` → 87 tests
+- `script_producer/` → 47 tests (subset of original)
+
+**Total: 1,197 tests as of 2026-05-28.**
 
 **Sanity checks** while running:
 ```bash
@@ -48,7 +54,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \
 
 ```
                                       ┌─────────────────────────────────────────┐
-                                      │        HeyGen Avatar Realtime API        │
+                                      │            HeyGen Canary API             │
                                       │  (text in → audio + HLS video out)       │
                                       └─────┬───────────────────────────┬───────┘
                                             ▲                           │
@@ -95,7 +101,7 @@ sqlite3 /tmp/showrunner_autonomous_loop.sqlite \
 | [`showrunner/`](showrunner/) | LLM scene planner → browser_executor (Playwright) + word_tracker (cue scheduler). | yes |
 | [`script_producer/`](script_producer/) | Library import only. 5 files used by showrunner: `word_tracker.py`, `markers.py`, `cues.py`, `config.py`, `persistence.py`. Full M2 daemon was dropped. | no |
 
-Not included: `activity_screen/` (M3 PNG screenshotter, unused), the full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), and HeyGen's backend service source.
+Dropped from the extraction: `activity_screen/` (M3 PNG screenshotter, unused), full M2 daemon (Claude scripts, GitHub fetcher, tier router, prompts, seeds), HeyGen monorepo source (`movio/`, `heygen/`, `lib/`).
 
 ---
 
@@ -126,7 +132,7 @@ At `fire_wall_ms`, POST `/producer/marker_fired` → broadcast via `/activity/se
 
 **Why this works**: both `word.start` and `currentTime` are on the same axis. Anchor = the map to wall clock. Stalls freeze `currentTime`; re-anchor 250 ms later shifts `wall_t0` forward by the stall amount.
 
-**Pipeline latency** (measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s).
+**Pipeline latency** (canary, measured): ~9-12 s from M1's `session.append_text()` → audible in HLS player. Made of: Redis XREAD (~0.2s) + TTS (1-3s) + inference (3-5s per 3.5s chunk) + KVS fragment + HLS segment + player buffer (~2s).
 
 ---
 
@@ -156,7 +162,7 @@ LLM authors `[CUE: {"op":"navigate","target":"https://github.com/foo/bar","lead_
 | 1 | Emitter's `word.start` is accurate | ✅ HeyGen's responsibility, stable |
 | 2 | `<video>.currentTime` is the real HLS playhead | ✅ |
 | 3 | Anchor is fresh (re-sampled recently) | ✅ 250 ms + event-driven |
-| 4 | HeyGen's pipeline keeps up (runway stays positive) | ⚠️ improved by disabling M1 paused keepalive; still needs live verification |
+| 4 | Canary's pipeline keeps up (runway stays positive) | ⚠️ improved by disabling M1 paused keepalive; still needs live verification |
 | 5 | word_tracker uses *latest* anchor at fire time, not at schedule time | ❌ uses precomputed `asyncio.sleep` — P2 |
 
 When 4 or 5 fails, cues fire while the avatar is still speaking earlier content. Both are unfixed.
@@ -165,13 +171,13 @@ When 4 or 5 fails, cues fire while the avatar is still speaking earlier content.
 
 ## Open problems
 
-> **The P0–P5 problems below describe the SYNC LAYER** (HeyGen audio + player clock + cue marker pipeline) and remain accurate. For the **CONTENT layer** (episode authoring, transition cues, chat consumption, leaderboard, anti-repeat), see the design specs under [docs/superpowers/specs/](docs/superpowers/specs/).
+> **The P0–P5 problems below describe the SYNC LAYER** (HeyGen audio + player clock + cue marker pipeline) and remain accurate. For the **CONTENT layer** (episode authoring, transition cues, chat consumption, leaderboard, anti-repeat — Plans 1–8 + §7+§8), see the latest entry in [CHANGELOG.md](CHANGELOG.md) and the current "open work" section of [PLAN.md](PLAN.md).
 
 ### P0 — Pipeline overcommit. MITIGATED, NEEDS LIVE VERIFICATION.
 
 `runway_sec` goes negative. Audio in the player lags the global media timeline by 10-50 s. Cues fire on the global timeline. Mismatch visible.
 
-Concrete trigger from a real run: M1 pushed 124 s of text in 130 s of wall time (49 s scene + 7×6.4 s keepalives). HeyGen delivered audio at real-time rate, so 51 s of text was queued ahead of where the player actually was. Cues for words at media=86 s fired when the player was at media=~70 s.
+Concrete trigger from a real run: M1 pushed 124 s of text in 130 s of wall time (49 s scene + 7×6.4 s keepalives). Canary delivered audio at real-time rate, so 51 s of text was queued ahead of where the player actually was. Cues for words at media=86 s fired when the player was at media=~70 s.
 
 Current mitigation: in external-producer mode, a paused M1 feeder now sends **no** keepalive speech. If showrunner stops pushing, the HeyGen session may idle out instead of M1 growing a stale speech queue.
 
@@ -243,7 +249,7 @@ Sessions cap at ~3600 s server-side. M1 rotates seamlessly but re-fires `seed_te
 | File | What |
 |---|---|
 | [`config.toml:24`](stream_orchestrator/config.toml#L24) | `seed_text` literal — ~30 s of speech |
-| [`supervisor.py`](stream_orchestrator/stream_orchestrator/supervisor.py) | `_open_session()` passes `seed_text` to HeyGen on every create |
+| [`supervisor.py`](stream_orchestrator/stream_orchestrator/supervisor.py) | `_open_session()` passes `seed_text` to canary on every create |
 
 ---
 
@@ -304,12 +310,12 @@ Single source of truth, every endpoint is listed in the docstring at the top:
 | `POST /producer/marker_fired` | word_tracker → M1 marker fire |
 | `GET /activity/segments` | SSE — marker fan-out to browser_pipeline |
 
-### Server-side reference (NOT in this repo)
+### Canary-side reference (NOT in this repo)
 
-These live in HeyGen's backend service, cited only for context when debugging server behavior:
-- request/response shapes (Avatar Realtime v3 DTO)
-- TTS → word timestamps (realtime emitter activity)
-- HLS URL builder, uses `PlaybackMode=LIVE_REPLAY`
+These are in the HeyGen monorepo, cited only for context when debugging server behavior:
+- `movio/api_service/app/api_types/avatar_realtime_v3_dto.py` — request/response shapes
+- `heygen/temporal/activity/avatar_realtime_emitter.py` — TTS → word timestamps
+- `lib/streaming/client.py` — HLS URL builder, uses `PlaybackMode=LIVE_REPLAY`
 
 ---
 
@@ -324,7 +330,7 @@ These live in HeyGen's backend service, cited only for context when debugging se
 | `bootstrap_timeout_sec` | 180 | [config.toml:114](stream_orchestrator/config.toml#L114) | Wait for `status=streaming` before declaring failure. |
 | `max_consecutive_bootstrap_failures` | 3 | [config.toml:74](stream_orchestrator/config.toml#L74) | Process exits 1 after this many fails. |
 | `heygen_push_below_sec` | 20.0 | [buffer_policy.py:31](showrunner/showrunner/buffer_policy.py#L31) | Showrunner's normal pusher gate. Pushes when runway falls below. |
-| `producer_silence_force_push_sec` | 25.0 | [buffer_policy.py:33](showrunner/showrunner/buffer_policy.py#L33) | Force-push next ready segment before HeyGen's 30s producer-idle timeout. |
+| `producer_silence_force_push_sec` | 25.0 | [buffer_policy.py:33](showrunner/showrunner/buffer_policy.py#L33) | Force-push next ready segment before canary's 30s producer-idle timeout. |
 | `committed_target_sec` | 45.0 | [buffer_policy.py:29](showrunner/showrunner/buffer_policy.py#L29) | Showrunner's committed queue target. |
 | `draft_low_sec` | 120.0 | [buffer_policy.py:28](showrunner/showrunner/buffer_policy.py#L28) | Showrunner's planner-ahead target. |
 
@@ -338,19 +344,19 @@ These live in HeyGen's backend service, cited only for context when debugging se
 - **scene** / **segment**: ~30 s LLM-planned chunk = narration + browser plan + cues. Used interchangeably.
 - **committed queue**: scenes LLM-approved, sitting in showrunner SQLite, waiting for runway gate.
 - **green/yellow/red tier**: director's URL trust classification. Green = whitelist (github, wikipedia). Yellow = unknown → read-only. Red = banned (paypal, social).
-- **API endpoint**: the HeyGen API endpoint we point at, `api.heygen.com`.
-- **M1/M2/M3**: internal module numbering used throughout the codebase. M1 = `stream_orchestrator`. M2 = `script_producer` (minimal slice survives). M3 = `activity_screen` (not included).
+- **canary**: HeyGen API endpoint we point at, `api-canary.heygen.com`.
+- **M1/M2/M3**: original monorepo module numbering. M1 = `stream_orchestrator`. M2 = `script_producer` (minimal slice survives). M3 = `activity_screen` (dropped).
 
 ---
 
 ## Asks
 
-1. **P0 live verification**. With paused keepalive removed, rerun a real live show and confirm runway no longer goes negative during showrunner stalls.
+1. **P0 live verification**. With paused keepalive removed, rerun a real canary show and confirm runway no longer goes negative during showrunner stalls.
 2. **P2 fix**. Port the rAF/tick pattern from [captions.html](stream_orchestrator/stream_orchestrator/captions.html) into [word_tracker.py](script_producer/script_producer/word_tracker.py). Suggested: shared `latest_anchor` + 50ms tick loop reading pending queue.
 3. **P4 fix**. DOM-based blocker detection instead of substring. Right Playwright API for "modal blocking interaction now"? `page.locator("[role=dialog][aria-modal=true]:visible")`? z-index heuristic?
-4. **Architecture critique**. Showrunner pushes atomic 30s scenes. Caption lookahead = 30s. Worth refactoring to stream sentence-by-sentence within a scene?
+4. **Architecture critique**. Showrunner pushes atomic 30s scenes. Caption lookahead = 30s. Worth refactoring to stream sentence-by-sentence within a scene? See [STATUS.md](STATUS.md).
 5. **Cleanup**. Merge `word_tracker.py` + `markers.py` into `showrunner/`, drop `script_producer/`. Mechanical, ~1-2h.
 
 ---
 
-*See [README.md](README.md) for quick-start, prerequisites, and the full test-suite commands.*
+*Standalone-repo test counts (2026-06-04): stream_orchestrator 213 · showrunner 1362 (+7 deselected live) · chat_ingester 114 · script_producer 53. See [README.md](README.md) for quick-start and [CHANGELOG.md](CHANGELOG.md) for history.*
diff --git a/chat_ingester/.python-version b/chat_ingester/.python-version
@@ -0,0 +1 @@
+3.11
diff --git a/chat_ingester/README.md b/chat_ingester/README.md
@@ -69,10 +69,10 @@ script accepts either). Tokens last ~60 days; refresh-flow is a v1.1 TODO.
 ### 3. Run it
 
 ```bash
-export TWITCH_BOT_TOKEN='oauth:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
+export TWITCH_BOT_TOKEN='oauth:<token>'
 export TWITCH_BOT_NICK='myreviewbot'
 export TWITCH_CHANNEL='yourchannelname'
-export ANTHROPIC_API_KEY='sk-ant-...'        # same key the showrunner uses
+export ANTHROPIC_API_KEY='<anthropic-api-key>'        # same key the showrunner uses
 scripts/05-run-chat-ingester.sh
 ```
 

diff --git a/chat_ingester/chat_ingester/queue_store.py b/chat_ingester/chat_ingester/queue_store.py
@@ -205,6 +205,11 @@ def open(self) -> None:
         # Verified during showrunner integration tests.
         conn.execute("PRAGMA journal_mode=WAL")
         conn.execute("PRAGMA synchronous=NORMAL")
+        # Wait up to 5s for a lock rather than failing instantly. The writer
+        # is the source of contention for the showrunner's readers; giving it
+        # (and any reader sharing this setup) a busy_timeout lets WAL
+        # checkpoints and concurrent access wait briefly instead of erroring.
+        conn.execute("PRAGMA busy_timeout=5000")
         conn.executescript(_SCHEMA_SQL)
         conn.commit()
         self._conn = conn