refactor(paper): add source-first Paper Flow V1#120
Merged
Conversation
6 tasks
…mmary map-reduce Rework Paper Flow V1 so identity and orchestration sit at the right altitude, without changing the source-first data model, stored schema, or the public paper_flow(input, *, cfg) -> PaperFlowResult signature. Identity vs translation: - Add PaperSourceRevision.from_parsed / PaperChunkSet.from_parsed_chunks / PaperGlobalSummary.from_draft smart constructors that mint every ID, content and producer hash, and resolve/validate citations internally. - flows/paper.py now imports zero private _paper_* helpers and computes no ID. It only fetches, parses, reads asset bytes, and maps preprocess/rag values to knowledge-native inputs. `knowledge` stays an import-linter leaf. Summarization: - Replace the manager/worker coordinator + Agent.as_tool() + hand-rolled concurrency-safe _SummaryBudget with a deterministic map-reduce: code tiles the chunk set (coverage guaranteed by construction), fans out one research agent per group via asyncio.gather + Semaphore, and runs one reducer. - Bounds are delegated to the SDK (ModelSettings.max_tokens, output_type) and asyncio (wait_for). PaperSummaryProducer identity becomes map-reduce-v1 with research_group_size; drop the six budget knobs and two validator clauses from PaperFlowCfg, add summary_research_group_size / summary_concurrency. Tests and docs: - Rebuild tests/paper_helpers.build_paper_result on the new constructors and rewrite the summary tests as map-reduce tests; update the e2e verifier and the paper example. - Document the identity-vs-translation and agentic-vs-deterministic-map-reduce principles in contexts/design/operations/orchestration.md and refresh the paper flow/knowledge design pages. verify.sh: ruff, basedpyright (0 errors), 7 import contracts, 334 tests, 84% cov. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
keli-wen
marked this pull request as ready for review
July 18, 2026 18:08
This was referenced Jul 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Paper(TreeKnowledge)output with a source-firstPaperFlowResultcontaining an immutable exact source revision, a deterministic page-awarePaperChunkSet, and an independently versioned citedPaperGlobalSummary.Agent.as_tool()research specialist, permits bounded parallel and overlapping follow-up calls, and rejects incomplete source coverage.list[str]paper search filter with the Pydantic-validatedPaperArtifactKindenum. Python callers usePaperArtifactKind.GLOBAL_SUMMARYorPaperArtifactKind.CHUNK_SET; JSON and YAML retain the corresponding string values while unknown kinds fail validation.put_paper(); selective embedding reuse; reopen-safe locator resolution; cross-record integrity checks; and a testedLegacyPapercompatibility boundary for existing databases.The cross-flow placement and reuse policy for Pydantic input/config types is intentionally tracked separately in #121 instead of being decided implicitly by this paper refactor.
Related Issues
Closes #119
Breaking Changes
paper_flow()now returnsPaperFlowResultand supports PDF-backed arXiv, HTTP, and local inputs in V1; raw text, non-PDF content, and unresolved DOI inputs fail explicitly.Papertree is no longer exported or generated. Existing version-2 databases and the bundled compatibility example load that shape asLegacyPaper.SemanticQuery.artifact_kindsnow validates aslist[PaperArtifactKind]; valid serialized string values remainpaper_summaryandpaper_chunk_set.Verification
./scripts/verify.sh— PASS: ruff format/check, basedpyright with 0 errors and warnings, all 7 import contracts, 335 tests, and 83.69% coverage..venv/bin/python scripts/verify_pdf_rag_e2e.py— live PASS against exact arXiv revision1706.03762v7: manager/research-agent orchestration, complete coverage of 36 chunks, 15 text pages, 19 source assets, 42 chunk-to-asset references, cited summary across 7 pages, summary retrieval, multi-head-attention passages in the top five, close/reopen, repeated searches, and canonical locator resolution.ciPASS; thepaper-flowjob PASS with its live step explicitly skipped because the repository has noOPENAI_API_KEY; the unrelatednewsjob failed on both the initial run and one retry because PR Newswire returned404for its RSS endpoint while the discovery and ticker-hint checks passed.Checklist
type(scope): summary../scripts/verify.shpasses.