docs: search plumbing for agent autoresearch loops - #279
Conversation
Add an overview chapter tying budgets, batch APIs, compact DerivedResult envelopes, and claim graphs together; expand Sphinx/README/getting-started with the same surface. Co-authored-by: Cursor <cursoragent@cursor.com>
📝 WalkthroughWalkthroughThe PR adds documentation for workload APIs, result serialization, budgets, batch evaluation, certificate handling, and autoresearch or agent-loop workflows across Sphinx, mdBook, the README, and the changelog. ChangesSearch Plumbing Documentation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/mdbook/src/intro.md`:
- Around line 17-23: Update the “Stateless by design” determinism claim to
distinguish reproducibility from strict determinism: explain that explicit
context improves reproducibility, step and seed controls can be deterministic,
and wall-clock budgets such as wall_ms remain best-effort because they depend on
machine load and scheduling.
In `@docs/mdbook/src/search-plumbing.md`:
- Around line 39-43: Update the failure-handling bullets in search-plumbing.md
to distinguish direct-call handling from integrate_many’s batch-result handling:
BudgetExceededError from batch candidates is represented as BatchItem(ok=False,
error=…) rather than caught by the loop. Qualify the slot-preservation statement
to note that ordinary failures become BatchItem entries while KeyboardInterrupt
and SystemExit propagate, consistent with batch.md.
- Around line 31-32: Update the code around item.value.to_dict(mode="compact")
to retain the compact envelope rather than assigning it to the discard variable.
Store the resulting payload in the collection, sink, or variable consumed by the
next iteration or referee, preserving the existing compact serialization mode.
In `@docs/sphinx/api/errors.rst`:
- Around line 172-178: Update the BudgetExceededError documentation to include
the run_with_wall_fallback timeout path, stating that it raises E-BUDGET-001
when the Python wall-clock fallback reaches its timeout without requiring a
cooperative engine checkpoint.
In `@docs/sphinx/api/workload.rst`:
- Around line 50-55: Update the documented run_with_wall_fallback signature to
require the keyword-only budget argument, matching the runtime definition in
run_with_wall_fallback and removing the misleading None default.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 27f20d18-4544-488c-88cd-59a2542cee6d
📒 Files selected for processing (15)
CHANGELOG.mdREADME.mddocs/mdbook/src/SUMMARY.mddocs/mdbook/src/batch.mddocs/mdbook/src/budgets.mddocs/mdbook/src/claim-graphs.mddocs/mdbook/src/derivations.mddocs/mdbook/src/getting-started.mddocs/mdbook/src/intro.mddocs/mdbook/src/python-api.mddocs/mdbook/src/search-plumbing.mddocs/sphinx/api/core.rstdocs/sphinx/api/errors.rstdocs/sphinx/api/workload.rstdocs/sphinx/index.rst
| **Agent loops.** Budgets and cancellation, batch APIs that never abort on one bad candidate, versioned compact result envelopes, and session-level [claim graphs](./claim-graphs.md) are first-class — see [Autoresearch / agent loops](./search-plumbing.md). | ||
|
|
||
| ## Design principles | ||
|
|
||
| **Explicit representations.** The type system distinguishes `UniPoly` (FLINT-backed univariate polynomial), `MultiPoly` (sparse multivariate), `RationalFunction`, and the generic `Expr` tree. Converting between them is an explicit call. There are no silent representation changes hiding performance cliffs. | ||
|
|
||
| **Stateless by design.** No global assumption contexts. No hidden caches that change behavior. All context (domains, simplification policy, precision) is passed explicitly or bundled into expression structure. This makes results deterministic and parallelism safe. | ||
| **Stateless by design.** No global assumption contexts. No hidden caches that change behavior. All context (domains, simplification policy, precision, budgets) is passed explicitly or scoped through `context(...)`. This makes results deterministic and parallelism safe. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Qualify the determinism claim for wall-clock budgets.
wall_ms depends on machine load and parallel scheduling. A seed does not make wall-clock cutoffs deterministic. State that explicit context improves reproducibility, while step and seed controls can be deterministic and wall-clock limits are best effort.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/mdbook/src/intro.md` around lines 17 - 23, Update the “Stateless by
design” determinism claim to distinguish reproducibility from strict
determinism: explain that explicit context improves reproducibility, step and
seed controls can be deterministic, and wall-clock budgets such as wall_ms
remain best-effort because they depend on machine load and scheduling.
| # Token-cheap record for the next iteration / a human referee | ||
| _ = item.value.to_dict(mode="compact") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Keep the compact envelope instead of discarding it.
_ = item.value.to_dict(mode="compact") creates the payload and immediately drops it. This does not create a record for the next iteration or a referee. Store the payload in a list, log sink, or variable used by the next step.
Proposed fix
+records = []
with ak.research.session(title="Sweep", pool=pool, capture=True) as s:
with ak.context(pool=pool, budget=ak.Budget(wall_ms=200, max_steps=50_000, seed=7)):
for item in ak.integrate_many(candidates, x, parallel=True):
if not item.ok:
continue
- _ = item.value.to_dict(mode="compact")
+ records.append(item.value.to_dict(mode="compact"))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/mdbook/src/search-plumbing.md` around lines 31 - 32, Update the code
around item.value.to_dict(mode="compact") to retain the compact envelope rather
than assigning it to the discard variable. Store the resulting payload in the
collection, sink, or variable consumed by the next iteration or referee,
preserving the existing compact serialization mode.
| - A **budget trip is a fine answer**, not a crash — catch `BudgetExceededError` | ||
| (`E-BUDGET-*`) and deprioritize that candidate. | ||
| - A **batch never drops a slot** — failures become `BatchItem(ok=False, error=…)`. | ||
| - **Compact mode never hides verification status** — `verification["status"]` | ||
| stays readable; Lean source is omitted on purpose. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Align the failure-handling rules with batch_map.
integrate_many uses the batch path, so candidate BudgetExceededError failures become BatchItem(ok=False, error=...); the loop does not catch the exception directly. Also, a batch does not “never” drop a slot because KeyboardInterrupt and SystemExit propagate, as documented in docs/mdbook/src/batch.md lines 118-121. Distinguish direct-call handling from batch-result handling and qualify the slot-preservation claim.
Proposed wording
- A budget trip is a fine answer, not a crash — catch `BudgetExceededError`
- (`E-BUDGET-*`) and deprioritize that candidate.
- A batch never drops a slot — failures become `BatchItem(ok=False, error=…)`.
+ For direct calls, catch `BudgetExceededError` (`E-BUDGET-*`). For batch
+ calls, inspect `BatchItem.error` and deprioritize failed candidates.
+ Batch calls preserve a slot for ordinary `Exception` failures, but
+ `KeyboardInterrupt` and `SystemExit` still stop the batch.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - A **budget trip is a fine answer**, not a crash — catch `BudgetExceededError` | |
| (`E-BUDGET-*`) and deprioritize that candidate. | |
| - A **batch never drops a slot** — failures become `BatchItem(ok=False, error=…)`. | |
| - **Compact mode never hides verification status** — `verification["status"]` | |
| stays readable; Lean source is omitted on purpose. | |
| For direct calls, catch `BudgetExceededError` (`E-BUDGET-*`). For batch | |
| calls, inspect `BatchItem.error` and deprioritize failed candidates. | |
| Batch calls preserve a slot for ordinary `Exception` failures, but | |
| `KeyboardInterrupt` and `SystemExit` still stop the batch. | |
| - **Compact mode never hides verification status** — `verification["status"]` | |
| stays readable; Lean source is omitted on purpose. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/mdbook/src/search-plumbing.md` around lines 39 - 43, Update the
failure-handling bullets in search-plumbing.md to distinguish direct-call
handling from integrate_many’s batch-result handling: BudgetExceededError from
batch candidates is represented as BatchItem(ok=False, error=…) rather than
caught by the loop. Qualify the slot-preservation statement to note that
ordinary failures become BatchItem entries while KeyboardInterrupt and
SystemExit propagate, consistent with batch.md.
| .. exception:: BudgetExceededError | ||
|
|
||
| Code prefix ``E-BUDGET-*``. A cooperative budget or cancellation trip — | ||
| not a mathematical failure. Raised when an active | ||
| :class:`~alkahest.Budget` is exceeded (or :func:`~alkahest.request_cancel` | ||
| was called) at a checkpoint inside an engine that honors budgets | ||
| (notably :func:`~alkahest.integrate`). See the |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Document the Python wall-clock fallback path.
python/alkahest/_budget.py:145-201 also raises BudgetExceededError with E-BUDGET-001 when run_with_wall_fallback reaches its timeout. This path does not require a cooperative engine checkpoint. Add it to this exception description so callers know the complete error boundary.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/sphinx/api/errors.rst` around lines 172 - 178, Update the
BudgetExceededError documentation to include the run_with_wall_fallback timeout
path, stating that it raises E-BUDGET-001 when the Python wall-clock fallback
reaches its timeout without requiring a cooperative engine checkpoint.
| .. function:: run_with_wall_fallback(fn, *args, budget=None, **kwargs) | ||
|
|
||
| Python-layer wall-clock fallback for callables that cannot raise | ||
| :exc:`BudgetExceededError` through their own return type (e.g. | ||
| :func:`simplify`). Prefer ``context(budget=...)`` for engines that already | ||
| honor Rust cooperative checkpoints. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make budget required in the documented signature.
The runtime signature in python/alkahest/_budget.py:145-201 requires the keyword-only argument budget: Budget. Documenting budget=None advertises a call that raises TypeError before the function runs.
-.. function:: run_with_wall_fallback(fn, *args, budget=None, **kwargs)
+.. function:: run_with_wall_fallback(fn, *args, budget, **kwargs)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| .. function:: run_with_wall_fallback(fn, *args, budget=None, **kwargs) | |
| Python-layer wall-clock fallback for callables that cannot raise | |
| :exc:`BudgetExceededError` through their own return type (e.g. | |
| :func:`simplify`). Prefer ``context(budget=...)`` for engines that already | |
| honor Rust cooperative checkpoints. | |
| .. function:: run_with_wall_fallback(fn, *args, budget, **kwargs) | |
| Python-layer wall-clock fallback for callables that cannot raise | |
| :exc:`BudgetExceededError` through their own return type (e.g. | |
| :func:`simplify`). Prefer ``context(budget=...)`` for engines that already | |
| honor Rust cooperative checkpoints. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/sphinx/api/workload.rst` around lines 50 - 55, Update the documented
run_with_wall_fallback signature to require the keyword-only budget argument,
matching the runtime definition in run_with_wall_fallback and removing the
misleading None default.
Merging this PR will not alter performance
Comparing 
Footnotes
|
1 participant
Summary
search-plumbing.mdlinking budgets, batch, compact envelopes, claim graphs, and certificate coverage.api/workload.rstplusDerivedResult.to_dict/BudgetExceededError/context(budget=…)entries.Test plan
mdbook build docs/mdbooksphinx-build -W docs/sphinx …Made with Cursor
Summary by CodeRabbit