Skip to content

feat(mind): add PageIndex-style structure retrieval#125

Merged
keli-wen merged 9 commits into
masterfrom
feat/pageindex-structure-tree
Jul 21, 2026
Merged

feat(mind): add PageIndex-style structure retrieval#125
keli-wen merged 9 commits into
masterfrom
feat/pageindex-structure-tree

Conversation

@keli-wen

@keli-wen keli-wen commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds PageIndex-style structure retrieval as a self-contained artifact plus a pure-agentic reasoning retriever, and refreshes the design principles that shape both. After review the original "vectorless empty shell + library page-refill" design was replaced by a self-contained one, so the final shape is:

  • knowledge: a shared StructureTree base and a source-bound PaperStructureTree whose leaf nodes carry their own page-cited text (internal nodes stay navigational). from_draft mints all identity, links, and citations, and copies minimal provenance (ArtifactMeta: as_of + a light source ref + source_content_hash) that is kept out of id / content_hash, so identity stays reproducible and the tree is storable standalone.
  • flows: PaperFlow(cfg) is a config-bound flow — bind the cfg once, build(input) takes only the operand, and the cfg type selects the knowledge shape (PaperStructureCfgPaperStructureTree). paper_flow remains the compatibility function for the semantic chunk/summary shape. A pipeline is pure processing and binds no library.
  • library: stores a self-contained tree standalone via put(tree) / open_structure(id) (no source or chunk set required); SQLite schema v5 drops the artifact→source foreign key with a migration; resolve returns a node's stored content with no query-time refill.
  • mind: AgenticRetriever(RetrievalCfg) is pure-agentic — an LLM agent reasons over one self-contained structure and returns evidence values (content included; locator is optional provenance), with no library binding. Mechanical retrieval (semantic / BM25) stays in library / rag; a future hybrid path composes the two.
  • preprocess: deterministic outline signals (headings, table-of-contents pages, printed-to-physical page offset) feed the single bounded structuring draft call.
  • defaults / robustness: PaperStructureCfg and RetrievalCfg default to gpt-5.6-luna; AgenticRetriever wraps the SDK MaxTurnsExceeded in RetrievalError and raises RetrievalCfg.max_turns to 20 so agentic traversal has head-room.
  • design principles: contexts/design/mind/retrieval.md, operations/orchestration.md, operations/naming.md, AGENTS.md, CLAUDE.md, and the quantmind-dev skill now state: bind config (not the operand) for batch reproducibility; the cfg type selects shape/strategy (typed dispatch, not a class hierarchy); artifacts are self-contained (content + provenance); persistence and retrieval are downstream and retrieval returns values; and mind is the reasoning layer while rag / library are the mechanical retrieval plane.
  • observability: scripts/verify_structure_e2e.py is a live smoke that builds, persists, reopens, and retrieves under the default model, wired as a path-filtered, non-required structure job in .github/workflows/e2e.yml.

Related Issue

Part of #95; design context in #122. Live-smoke follow-ups filed: #126 (structured output fails on json_object-only providers) and #127 (surface token-usage / cost).

Verification

  • bash scripts/verify.sh — pass: Ruff format + lint, basedpyright (0 errors), import-linter contracts, and 382 tests at 84.35% coverage.
  • Live structure smoke: python scripts/verify_structure_e2e.py — PASS with gpt-5.6-luna (self-contained tree covering every page, put / open_structure round-trip identical, agentic retrieve returns evidence).

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes.
  • Every applicable live-network component smoke test passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes.

keli-wen and others added 4 commits July 20, 2026 01:19
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
@keli-wen keli-wen added type: feature Adds a new capability or observable behavior area: mind Memory, tools, MCP integration, and the cognitive layer labels Jul 19, 2026
@keli-wen keli-wen self-assigned this Jul 20, 2026
keli-wen and others added 5 commits July 20, 2026 20:47
Redesign the PageIndex structure-tree feature around one altitude rule:
pipelines produce complete, self-contained artifacts, and persistence and
retrieval are separate downstream concerns.

- knowledge: PaperStructureTree leaf nodes now carry their own page text
  (internal nodes stay navigational); the "nodes must not copy content"
  rule is inverted to require leaf content. content_hash covers it, so
  re-runs stay idempotent.
- library: add open_structure() to load a tree by id; resolve() returns a
  node's stored content instead of refilling it from the source; a tree
  persists and reopens as an identical value with no chunk set present.
- mind: replace StructureRetriever(library=...) with the library-free
  function retrieve(tree, question, *, cfg) returning self-contained
  evidence values, with locator as optional provenance only.
- flows: introduce the PaperFlow document handle in a flows/paper/ package
  (open() parses once; build_structure() / extract_knowledge() are pure
  derivations); keep paper_flow as a thin compatibility wrapper.
- example: build -> retrieve in memory (no library); persist/reopen shown
  as an optional dump/load-symmetry section.
- docs & philosophy: rewrite contexts/design/mind/retrieval.md; add the
  pipeline-vs-component altitude, callable-shape, and immutable
  document-handle guidance to orchestration.md, naming.md, AGENTS.md,
  CLAUDE.md, and the quantmind-dev skill; refresh docs catalog rows.

scripts/verify.sh green: 375 tests, 84.49% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…able tree

Reshape the structure-retrieval API to the reviewed design principles and carry
those principles into the repo guidance.

Principles (docs): bind config at construction for batch reproducibility and
pass only the operand per call; the cfg *type* selects the shape/strategy
(typed dispatch, not a class hierarchy); a self-contained artifact carries its
own content plus minimal provenance metadata so it stores standalone; retrieval
returns values while persistence/retrieval stay downstream. Updated
orchestration.md, naming.md, mind/retrieval.md, AGENTS.md, CLAUDE.md, and the
quantmind-dev skill; replaced the earlier document-handle guidance.

Code:
- knowledge: PaperStructureTree mixes in a light ArtifactMeta provenance base
  (as_of + source ref + source_content_hash), populated by from_draft and kept
  out of id/content_hash so identity stays reproducible.
- library: put() stores a self-contained tree standalone (no source or chunk
  set); schema v5 drops the artifact->source FK with a migration; open_structure
  and content-carrying resolve unchanged.
- mind: Retrieve(cfg) class dispatching by cfg type; RetrievalCfg base +
  AgenticRetrievalCfg(mode="tree"); single-pass path dropped; no library import.
- flows: PaperFlow(cfg).build(input) config-bound flow, cfg type selecting the
  knowledge shape; open()/build_structure()/extract_knowledge() removed;
  paper_flow kept as the semantic-shape compatibility wrapper.
- example rewritten to build -> retrieve in memory, optional put(tree)/reopen.
- example bundle DB migrated v4 -> v5 to match the schema bump (kept at v5 so
  opening it triggers no in-place migration write).

scripts/verify.sh green: 384 tests, 84.38% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename Retrieve -> AgenticRetriever and drop the strategy-dispatch shell: mind
is the reasoning layer (an LLM agent reasons over a structure), and mechanical
retrieval (semantic vector search / BM25) belongs to library / rag, so there is
no agentic/semantic/hybrid cfg-type dispatch here.

- mind: Retrieve(cfg) -> AgenticRetriever(cfg); retrieve(retrievable, question)
  over a Retrievable (StructureTree today, widening within knowledge to graph,
  never a vector store); library-free, evidence values with optional locator.
- configs: collapse the RetrievalCfg base + AgenticRetrievalCfg subclass into a
  single RetrievalCfg; drop the `mode` field (structure kind is the operand's).
- charter: mind = pure-agentic reasoning layer vs rag/library = mechanical
  retrieval plane, into AGENTS.md, CLAUDE.md, retrieval.md, orchestration.md,
  naming.md, and the quantmind-dev skill.
- example, catalog (docs/README.md, docs/library.md), and tests updated.

scripts/verify.sh green: 382 tests, 84.37% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address gaps a live smoke surfaced that offline verify.sh hides by mocking the
SDK model:

- default model: PaperStructureCfg + RetrievalCfg default to gpt-5.6-luna, which
  converges the agentic traversal in ~3 turns (vs gpt-4o-mini's 10+) and is
  cheaper. Other flows (paper_flow / news / earnings / magic) are unchanged.
- retrieve robustness: wrap the SDK MaxTurnsExceeded into RetrievalError with an
  actionable message; raise RetrievalCfg.max_turns default 10 -> 20 so agentic
  traversal has head-room before it fails closed.
- live e2e smoke: scripts/verify_structure_e2e.py builds a self-contained tree
  from the golden fixture PDF under the default model, dumps and reopens it via
  the library unchanged, and runs a real AgenticRetriever traversal — exactly the
  path offline tests mock away. Wired as a path-filtered, non-required `structure`
  job in e2e.yml plus a docs/README.md catalog row.

Verified live (OpenAI, gpt-5.6-luna): scripts/verify_structure_e2e.py -> PASS
(6-node tree covering all pages, put/open round-trip identical, 3 evidence
nodes). scripts/verify.sh green offline: 382 tests, 84.35% coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@keli-wen
keli-wen marked this pull request as ready for review July 21, 2026 04:07
@keli-wen
keli-wen merged commit a10eb2b into master Jul 21, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: mind Memory, tools, MCP integration, and the cognitive layer type: feature Adds a new capability or observable behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant