feat(mind): add PageIndex-style structure retrieval#125
Merged
Conversation
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Redesign the PageIndex structure-tree feature around one altitude rule: pipelines produce complete, self-contained artifacts, and persistence and retrieval are separate downstream concerns. - knowledge: PaperStructureTree leaf nodes now carry their own page text (internal nodes stay navigational); the "nodes must not copy content" rule is inverted to require leaf content. content_hash covers it, so re-runs stay idempotent. - library: add open_structure() to load a tree by id; resolve() returns a node's stored content instead of refilling it from the source; a tree persists and reopens as an identical value with no chunk set present. - mind: replace StructureRetriever(library=...) with the library-free function retrieve(tree, question, *, cfg) returning self-contained evidence values, with locator as optional provenance only. - flows: introduce the PaperFlow document handle in a flows/paper/ package (open() parses once; build_structure() / extract_knowledge() are pure derivations); keep paper_flow as a thin compatibility wrapper. - example: build -> retrieve in memory (no library); persist/reopen shown as an optional dump/load-symmetry section. - docs & philosophy: rewrite contexts/design/mind/retrieval.md; add the pipeline-vs-component altitude, callable-shape, and immutable document-handle guidance to orchestration.md, naming.md, AGENTS.md, CLAUDE.md, and the quantmind-dev skill; refresh docs catalog rows. scripts/verify.sh green: 375 tests, 84.49% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…able tree Reshape the structure-retrieval API to the reviewed design principles and carry those principles into the repo guidance. Principles (docs): bind config at construction for batch reproducibility and pass only the operand per call; the cfg *type* selects the shape/strategy (typed dispatch, not a class hierarchy); a self-contained artifact carries its own content plus minimal provenance metadata so it stores standalone; retrieval returns values while persistence/retrieval stay downstream. Updated orchestration.md, naming.md, mind/retrieval.md, AGENTS.md, CLAUDE.md, and the quantmind-dev skill; replaced the earlier document-handle guidance. Code: - knowledge: PaperStructureTree mixes in a light ArtifactMeta provenance base (as_of + source ref + source_content_hash), populated by from_draft and kept out of id/content_hash so identity stays reproducible. - library: put() stores a self-contained tree standalone (no source or chunk set); schema v5 drops the artifact->source FK with a migration; open_structure and content-carrying resolve unchanged. - mind: Retrieve(cfg) class dispatching by cfg type; RetrievalCfg base + AgenticRetrievalCfg(mode="tree"); single-pass path dropped; no library import. - flows: PaperFlow(cfg).build(input) config-bound flow, cfg type selecting the knowledge shape; open()/build_structure()/extract_knowledge() removed; paper_flow kept as the semantic-shape compatibility wrapper. - example rewritten to build -> retrieve in memory, optional put(tree)/reopen. - example bundle DB migrated v4 -> v5 to match the schema bump (kept at v5 so opening it triggers no in-place migration write). scripts/verify.sh green: 384 tests, 84.38% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename Retrieve -> AgenticRetriever and drop the strategy-dispatch shell: mind is the reasoning layer (an LLM agent reasons over a structure), and mechanical retrieval (semantic vector search / BM25) belongs to library / rag, so there is no agentic/semantic/hybrid cfg-type dispatch here. - mind: Retrieve(cfg) -> AgenticRetriever(cfg); retrieve(retrievable, question) over a Retrievable (StructureTree today, widening within knowledge to graph, never a vector store); library-free, evidence values with optional locator. - configs: collapse the RetrievalCfg base + AgenticRetrievalCfg subclass into a single RetrievalCfg; drop the `mode` field (structure kind is the operand's). - charter: mind = pure-agentic reasoning layer vs rag/library = mechanical retrieval plane, into AGENTS.md, CLAUDE.md, retrieval.md, orchestration.md, naming.md, and the quantmind-dev skill. - example, catalog (docs/README.md, docs/library.md), and tests updated. scripts/verify.sh green: 382 tests, 84.37% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address gaps a live smoke surfaced that offline verify.sh hides by mocking the SDK model: - default model: PaperStructureCfg + RetrievalCfg default to gpt-5.6-luna, which converges the agentic traversal in ~3 turns (vs gpt-4o-mini's 10+) and is cheaper. Other flows (paper_flow / news / earnings / magic) are unchanged. - retrieve robustness: wrap the SDK MaxTurnsExceeded into RetrievalError with an actionable message; raise RetrievalCfg.max_turns default 10 -> 20 so agentic traversal has head-room before it fails closed. - live e2e smoke: scripts/verify_structure_e2e.py builds a self-contained tree from the golden fixture PDF under the default model, dumps and reopens it via the library unchanged, and runs a real AgenticRetriever traversal — exactly the path offline tests mock away. Wired as a path-filtered, non-required `structure` job in e2e.yml plus a docs/README.md catalog row. Verified live (OpenAI, gpt-5.6-luna): scripts/verify_structure_e2e.py -> PASS (6-node tree covering all pages, put/open round-trip identical, 3 evidence nodes). scripts/verify.sh green offline: 382 tests, 84.35% coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 20, 2026
keli-wen
marked this pull request as ready for review
July 21, 2026 04:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds PageIndex-style structure retrieval as a self-contained artifact plus a pure-agentic reasoning retriever, and refreshes the design principles that shape both. After review the original "vectorless empty shell + library page-refill" design was replaced by a self-contained one, so the final shape is:
StructureTreebase and a source-boundPaperStructureTreewhose leaf nodes carry their own page-cited text (internal nodes stay navigational).from_draftmints all identity, links, and citations, and copies minimal provenance (ArtifactMeta:as_of+ a light source ref +source_content_hash) that is kept out ofid/content_hash, so identity stays reproducible and the tree is storable standalone.PaperFlow(cfg)is a config-bound flow — bind the cfg once,build(input)takes only the operand, and the cfg type selects the knowledge shape (PaperStructureCfg→PaperStructureTree).paper_flowremains the compatibility function for the semantic chunk/summary shape. A pipeline is pure processing and binds no library.put(tree)/open_structure(id)(no source or chunk set required); SQLite schemav5drops the artifact→source foreign key with a migration;resolvereturns a node's stored content with no query-time refill.AgenticRetriever(RetrievalCfg)is pure-agentic — an LLM agent reasons over one self-contained structure and returns evidence values (content included;locatoris optional provenance), with no library binding. Mechanical retrieval (semantic / BM25) stays inlibrary/rag; a future hybrid path composes the two.PaperStructureCfgandRetrievalCfgdefault togpt-5.6-luna;AgenticRetrieverwraps the SDKMaxTurnsExceededinRetrievalErrorand raisesRetrievalCfg.max_turnsto 20 so agentic traversal has head-room.contexts/design/mind/retrieval.md,operations/orchestration.md,operations/naming.md,AGENTS.md,CLAUDE.md, and thequantmind-devskill now state: bind config (not the operand) for batch reproducibility; the cfg type selects shape/strategy (typed dispatch, not a class hierarchy); artifacts are self-contained (content + provenance); persistence and retrieval are downstream and retrieval returns values; andmindis the reasoning layer whilerag/libraryare the mechanical retrieval plane.scripts/verify_structure_e2e.pyis a live smoke that builds, persists, reopens, and retrieves under the default model, wired as a path-filtered, non-requiredstructurejob in.github/workflows/e2e.yml.Related Issue
Part of #95; design context in #122. Live-smoke follow-ups filed: #126 (structured output fails on
json_object-only providers) and #127 (surface token-usage / cost).Verification
bash scripts/verify.sh— pass: Ruff format + lint, basedpyright (0 errors), import-linter contracts, and 382 tests at 84.35% coverage.python scripts/verify_structure_e2e.py— PASS withgpt-5.6-luna(self-contained tree covering every page,put/open_structureround-trip identical, agentic retrieve returns evidence).Checklist
type(scope): summary.bash scripts/verify.shpasses.