Reuse parent prompt cache for asynchronous token-budget summaries

[furgalep]

Published head: 44873c71. Rebased on the updated #319/#318. Summarizer changes are unchanged. Agents/runtime/UnifiedLLM regression suite: 2,098 passed, 3 skipped.

Summary

Make token-budget summarization an asynchronous branch of the parent's completed LLM request. It reuses the same rendered messages, tools, client, sampling settings and cache-routing key, then appends a short summary instruction.

This is a conversation fork, not a clone of the Python agent. No new UnifiedLLM abstraction or provider-name rules are added.

Stack: based on #319 (which depends on #318); parallel to #320.

Why

The standalone summarizer renders history again as Markdown under a different prompt. That history cannot reuse the parent's cached prefix. The middleware boundary already has the request that was sent, so it provides the smallest reuse point without another renderer pass or a stored-request interface.

The parent keeps running. The fork sees only events present before that request; the new response and tool work remain active. A completed summary is applied at the existing BeforeTurn boundary, and only if its source event IDs still match.

Code walkthrough — what changed and why

1. agents/summarization.py: TokenBudgetSummarizer installs an LLM middleware handler. Provider usage triggers one background fork; a task-local guard prevents recursive forks. A small container-copy helper detaches only dicts/lists and borrows tools, clients, response objects and cache boundaries. This protects asynchronous inputs without cloning tool owners or opaque state.
2. Summary result handling: keep the parent tool definitions and tool choice because changing them can invalidate caching. The fork executes no tools. It accepts prose or exactly one return_result whose result is a nonempty string, decoded as data. An unusable model reply gets one standalone attempt; provider/policy exceptions leave history untouched rather than taking that fallback.
3. runtime/actor.py and runtime/middleware.py: expose the effective client and filtered-history flag on LLMCallContext; retain the effective cache key in its params after dispatch. This includes method-specific model overrides without inspecting strategy/provider internals in the summarizer. Tests verify installing the summarizer does not change the parent request.
4. Configuration and model switching: TokenBudgetConfig.reuse_parent_prefix=True is the default. False, an explicitly different summarizer client, filtered history, or structured output uses standalone summarization. Model-limit updates preserve that setting. MethodSummarizer remains standalone.
5. Shutdown: SummarizationAgent.aclose() unsubscribes and awaits task cancellation. CodingAgent closes summarizers before their shared client, so shutdown cannot leave a background call using a closed pool.
6. Tests and experiment: regression tests cover async progress, one pending task, middleware/recursion, ownership, stale ranges, failures/fallback and shutdown. Mocked SDK transports compare the actual OpenAI/Anthropic request bodies. The experiment scripts measure cache reads through NVIDIA Inference Hub with trailing dynamic context.

Live evidence

One run per variant, NVIDIA Inference Hub, 2026-09-13:

Probe	OpenAI	Claude Sonnet
Protocol fork	5,772 / 5,831 cached (99.0%)	10,106 / 10,174 (99.3%)
Standalone comparison	0 / 5,788 (0%)	0 / 10,124 (0%)
Installed CodeAct summarizer	7,141 / 7,214 (99.0%)	12,213 / 12,399 (98.5%)

Both installed summaries preserved the test decision, budget and owner and were applied successfully. An initial installed probe correctly rejected a CodeAct return_result reply; that led to the data-only decoding regression and fix.

Total live spend: 12 calls, 95,472 input tokens (including cached tokens), 786 output tokens. No additional paid rerun after the ownership/fallback/shutdown review fixes; their request shape is unchanged and checked offline.

Limits

• 99% is a measured result, not a guarantee. Prefix length, live suffix, provider routing/expiry and middleware edits affect cache reuse. A cached whole-history request may still cost more than a very small standalone range.
• The fork is one turn behind by design and uses the parent's rendered/truncated view. It does not change collapse semantics or promise atomic tool-batch collapse.
• Unsupported/failed fork replies may incur one standalone summary. Failures and stale results preserve history.
• No general agent-cloning API, new provider policy, UI controls, or MethodSummarizer redesign.

Validation

Wren accepted source head 23ac4824 after independent review and 2,473 scoped tests. Regressions were observed failing before fixes. Parent-request parity, actual SDK wire prefixes, bound-tool ownership and nested-container isolation are covered. Ruff and whitespace checks pass.

Full offline rerun on 23ac4824: 7,699 passed, 6 skipped, 311 deselected, 3 expected failures, in 266 seconds. Published head 1f7023db adds only the final validation paragraph to the experiment README.

Design, reproduction commands, and results.

Summary by CodeRabbit

• New Features

  • Token-budget summarization can reuse completed parent requests to reduce duplicate work.
  • Added configuration to control parent-prefix reuse, with standalone fallback support.
  • Summarization validates results and preserves conversation history when processing fails.
  • Summary documents now explain how to recover archived and nested source events.
  • Runtime middleware exposes effective client and filtered-history context for request handling.

• Bug Fixes

  • Improved asynchronous shutdown for summarizers and coding agents.
  • Preserved existing summarizer settings when applying model token limits.

• Documentation

  • Added guidance covering summarization design, usage, fallback behavior, testing, and measured results.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7a5ba597-ef14-42ef-8f5d-6cc1dae86d12

📥 Commits

Reviewing files that changed from the base of the PR and between 1f7023d and 96a74d8.

📒 Files selected for processing (2)

• src/nooa/runtime/event_manager.py
• tests/runtime/test_events_api.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds asynchronous provider-request forking for token-budget summarization. It preserves parent requests, validates fork results, supports standalone fallback, exposes runtime context fields, adds shutdown handling, improves summary recovery instructions, and provides provider probes and tests.

Changes

Forked summarization

Layer / File(s)	Summary
Runtime contracts and configuration src/nooa/config/summarizer_config.py, src/nooa/runtime/*, src/nooa/interactive.py	Adds reuse_parent_prefix, exposes the effective client and filtered-history state, initializes cache keys before middleware, and preserves summarizer configuration during model-limit updates.
Fork execution and result handling src/nooa/agents/summarization.py, tests/agents/test_forked_summarizer.py, tests/agents/test_forked_summarizer_wire.py, tests/agents/test_summarization_agents.py	Adds isolated asynchronous forks, tool restrictions, result validation, standalone fallback, stale-result protection, recursion prevention, cancellation, and provider-specific request tests.
Agent shutdown ordering packages/nooa-cli/src/nooa_cli/coding/agent.py, packages/nooa-cli/tests/test_coding_agent.py	Closes registered summarizers before the shared client and tests the cleanup order.
Summary recovery instructions src/nooa/runtime/event_manager.py, tests/runtime/test_events_api.py	Adds recovery instructions to collapsed summaries and tests their content in flat, nested, Markdown, and XML contexts.
Provider probes and experiment documentation experiments/forked-summarizer/*	Adds OpenAI and Anthropic probes for parent, forked, standalone, and runtime summarization requests. Documents the design, protocol, fallback behavior, shutdown handling, and measured results.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Agent
  participant Summarizer
  participant Middleware
  participant Provider
  Agent->>Summarizer: schedule token-budget summary
  Summarizer->>Middleware: run isolated fork request
  Middleware->>Provider: send parent-prefix request
  Provider-->>Middleware: return summary response
  Middleware-->>Summarizer: return validated summary or fallback signal
  Summarizer-->>Agent: apply summary if source events are unchanged

Loading

Suggested reviewers: atharva-kanherkar

Merge Risk: ⚪ Minimal · up to 43291

The summary recovery guidance is covered by regression tests and does not leave an actionable merge risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 24.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 75 functions across 14 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reusing the parent prompt cache for asynchronous token-budget summaries.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cache-sharing-summarizer

---

_{Comment @coderabbitai help to get the list of available commands.}

[furgalep]

Added deterministic archive-recovery instructions to Summary.doc (96a74d8): search including archived events with a bounded result count, read an original event by tag, and expand this summary or nested summaries. The LLM does not generate these instructions. Four new regression cases failed before the change; 88 targeted tests pass afterward, including rendered hints and collapse/resume coverage. Ruff and formatting pass. Sent the delta to Wren for re-review.