refactor(paper): add source-first Paper Flow V1 (#120)

## Summary

- Replaces the model-authored `Paper(TreeKnowledge)` output with a
source-first `PaperFlowResult` containing an immutable exact source
revision, a deterministic page-aware `PaperChunkSet`, and an
independently versioned cited `PaperGlobalSummary`.
- Implements manager-style multi-agent summarization with the OpenAI
Agents SDK: one coordinator retains final-summary ownership, delegates
required chunk ranges through a real `Agent.as_tool()` research
specialist, permits bounded parallel and overlapping follow-up calls,
and rejects incomplete source coverage.
- Enforces shared research call, concurrency, turn, input-token,
worker-output, total-output, final-output, citation, and runtime limits.
Worker reports contain typed
context/contribution/method/result/limitation findings with
code-validated chunk/page coordinates.
- Replaces the open `list[str]` paper search filter with the
Pydantic-validated `PaperArtifactKind` enum. Python callers use
`PaperArtifactKind.GLOBAL_SUMMARY` or `PaperArtifactKind.CHUNK_SET`;
JSON and YAML retain the corresponding string values while unknown kinds
fail validation.
- Adds SQLite schema version 3 with explicit source/blob,
artifact/member, lineage, and search-projection tables; atomic
`put_paper()`; selective embedding reuse; reopen-safe locator
resolution; cross-record integrity checks; and a tested `LegacyPaper`
compatibility boundary for existing databases.
- Moves retrieval-text selection out of canonical knowledge models,
preserves existing non-paper search behavior, and documents the Source →
Artifact → SearchProjection contract across flow, knowledge, RAG,
library, examples, and contributor guidance.

The cross-flow placement and reuse policy for Pydantic input/config
types is intentionally tracked separately in #121 instead of being
decided implicitly by this paper refactor.

## Related Issues

Closes #119

- #121 tracks ownership, reuse, naming, supported-variant, schema, and
migration rules for typed operation inputs and stage configuration.

## Breaking Changes

- `paper_flow()` now returns `PaperFlowResult` and supports PDF-backed
arXiv, HTTP, and local inputs in V1; raw text, non-PDF content, and
unresolved DOI inputs fail explicitly.
- The former primary `Paper` tree is no longer exported or generated.
Existing version-2 databases and the bundled compatibility example load
that shape as `LegacyPaper`.
- `SemanticQuery.artifact_kinds` now validates as
`list[PaperArtifactKind]`; valid serialized string values remain
`paper_summary` and `paper_chunk_set`.

## Verification

- `./scripts/verify.sh` — PASS: ruff format/check, basedpyright with 0
errors and warnings, all 7 import contracts, 335 tests, and 83.69%
coverage.
- `.venv/bin/python scripts/verify_pdf_rag_e2e.py` — live PASS against
exact arXiv revision `1706.03762v7`: manager/research-agent
orchestration, complete coverage of 36 chunks, 15 text pages, 19 source
assets, 42 chunk-to-asset references, cited summary across 7 pages,
summary retrieval, multi-head-attention passages in the top five,
close/reopen, repeated searches, and canonical locator resolution.
- Focused config, flow, library, and live-verifier tests — PASS: 41
tests cover typed artifact parsing, real agent-tool construction, worker
concurrency and budgets, complete-coverage enforcement, persistence,
retrieval, and E2E assertions.
- Commit and pre-push hooks — PASS: file hygiene, YAML validation,
large-file check, ruff, and the deterministic verification harness.
- GitHub Actions — required `ci` PASS; the `paper-flow` job PASS with
its live step explicitly skipped because the repository has no
`OPENAI_API_KEY`; the unrelated `news` job failed on both the initial
run and one retry because PR Newswire returned `404` for its RSS
endpoint while the discovery and ticker-hint checks passed.

## Checklist

- [x] The title uses English Conventional Commit format: `type(scope):
summary`.
- [x] The related issue or design discussion is linked when applicable.
- [x] `./scripts/verify.sh` passes.
- [x] Every applicable live-network component smoke test passes, or this
PR states why none applies.
- [x] Public behavior has focused tests, an example, and documentation
where applicable.
- [x] The PR contains no unrelated changes.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: keli-wen

.agents/skills/quantmind-dev/references/develop-components.md

@@ -43,11 +43,13 @@ apply throughout.
 - Pydantic models, `frozen=True`, `extra="forbid"`.
 - Every `BaseKnowledge` subclass **must** require `as_of: datetime`
   (financial time-sensitivity is mandatory) and a typed `source: SourceRef`
-  (no bare strings), and **must** override `embedding_text()`.
+  (no bare strings).
+- Canonical models do not select retrieval text or store vectors. Put
+  rebuildable text projections in `quantmind.library`.
 - Pick one shape: `FlattenKnowledge` (atomic card), `TreeKnowledge`
-  (hierarchical artifact), or `GraphKnowledge` (placeholder). Whole-document
-  objects are `TreeKnowledge` even when a flatten card exists alongside
-  (e.g. `Paper` vs `PaperKnowledgeCard`).
+  (hierarchical artifact), or `GraphKnowledge` (placeholder) for conventional
+  knowledge. Source-first paper revisions and independently versioned paper
+  artifacts use their dedicated frozen models instead of `BaseKnowledge`.
 
 ### `quantmind/configs/` — operation cfg + typed inputs
 

.claude/skills/quantmind-dev/references/develop-components.md

@@ -43,11 +43,13 @@ apply throughout.
 - Pydantic models, `frozen=True`, `extra="forbid"`.
 - Every `BaseKnowledge` subclass **must** require `as_of: datetime`
   (financial time-sensitivity is mandatory) and a typed `source: SourceRef`
-  (no bare strings), and **must** override `embedding_text()`.
+  (no bare strings).
+- Canonical models do not select retrieval text or store vectors. Put
+  rebuildable text projections in `quantmind.library`.
 - Pick one shape: `FlattenKnowledge` (atomic card), `TreeKnowledge`
-  (hierarchical artifact), or `GraphKnowledge` (placeholder). Whole-document
-  objects are `TreeKnowledge` even when a flatten card exists alongside
-  (e.g. `Paper` vs `PaperKnowledgeCard`).
+  (hierarchical artifact), or `GraphKnowledge` (placeholder) for conventional
+  knowledge. Source-first paper revisions and independently versioned paper
+  artifacts use their dedicated frozen models instead of `BaseKnowledge`.
 
 ### `quantmind/configs/` — operation cfg + typed inputs
 

.github/workflows/e2e.yml

@@ -19,6 +19,11 @@ on:
       - 'quantmind/preprocess/fetch/arxiv.py'
       - 'quantmind/preprocess/format/pdf.py'
       - 'quantmind/rag/**'
+      - 'quantmind/configs/paper.py'
+      - 'quantmind/flows/_paper_summary.py'
+      - 'quantmind/flows/paper.py'
+      - 'quantmind/knowledge/paper.py'
+      - 'quantmind/library/**'
       - 'scripts/verify_pdf_rag_e2e.py'
       - 'pyproject.toml'
   schedule:
@@ -38,7 +43,7 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       news: ${{ steps.filter.outputs.news }}
-      pdf_rag: ${{ steps.filter.outputs.pdf_rag }}
+      paper_flow: ${{ steps.filter.outputs.paper_flow }}
 
     steps:
       - name: Checkout
@@ -60,9 +65,14 @@ jobs:
               - 'quantmind/preprocess/format/html.py'
               - 'quantmind/preprocess/fetch/**'
               - 'pyproject.toml'
-            pdf_rag:
+            paper_flow:
               - '.github/workflows/e2e.yml'
               - 'scripts/verify_pdf_rag_e2e.py'
+              - 'quantmind/configs/paper.py'
+              - 'quantmind/flows/_paper_summary.py'
+              - 'quantmind/flows/paper.py'
+              - 'quantmind/knowledge/paper.py'
+              - 'quantmind/library/**'
               - 'quantmind/preprocess/fetch/arxiv.py'
               - 'quantmind/preprocess/format/pdf.py'
               - 'quantmind/rag/**'
@@ -98,11 +108,13 @@ jobs:
       - name: Run live news E2E
         run: .venv/bin/python scripts/verify_news_e2e.py
 
-  pdf-rag:
+  paper-flow:
     needs: changes
-    if: github.event_name != 'pull_request' || needs.changes.outputs.pdf_rag == 'true'
+    if: github.event_name != 'pull_request' || needs.changes.outputs.paper_flow == 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 10
+    env:
+      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 
     steps:
       - name: Checkout
@@ -125,5 +137,17 @@ jobs:
       - name: Install project runtime dependencies
         run: uv pip install --python .venv/bin/python -e .
 
-      - name: Run live PDF RAG E2E
+      - name: Check OpenAI credential
+        id: openai
+        shell: bash
+        run: |
+          if [[ -n "${OPENAI_API_KEY}" ]]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "available=false" >> "$GITHUB_OUTPUT"
+            echo "::notice::Skipping Paper Flow V1 E2E because OPENAI_API_KEY is not configured."
+          fi
+
+      - name: Run live Paper Flow V1 E2E
+        if: steps.openai.outputs.available == 'true'
         run: .venv/bin/python scripts/verify_pdf_rag_e2e.py

README.md

@@ -156,7 +156,7 @@ We use [uv](https://github.com/astral-sh/uv) for fast and reliable Python packag
 
 Component-specific guides and architecture notes live under [`docs/`](docs/).
 
-#### Run a single paper through `paper_flow`
+#### Build an exact paper source, chunks, and cited summary
 
 ```python
 import asyncio
@@ -167,11 +167,12 @@ from quantmind.flows import paper_flow
 
 
 async def main() -> None:
-    paper = await paper_flow(
-        ArxivIdentifier(id="2401.12345"),
+    result = await paper_flow(
+        ArxivIdentifier(id="1706.03762v7"),
         cfg=PaperFlowCfg(model="gpt-4o-mini"),
     )
-    print(paper.model_dump_json(indent=2))
+    print(result.global_summary.summary)
+    print(result.source_revision.id, result.chunk_set.id)
 
 
 asyncio.run(main())
@@ -219,17 +220,17 @@ async def main() -> None:
         "Pull arXiv 2401.12345 about cross-sectional momentum; use gpt-4o-mini.",
         target_flow=paper_flow,
     )
-    paper = await paper_flow(inp, cfg=cfg)
-    print(paper.model_dump_json(indent=2))
+    result = await paper_flow(inp, cfg=cfg)
+    print(result.global_summary.summary)
 
 
 asyncio.run(main())
 ```
 
-> **Note**: QuantMind is mid-migration to OpenAI Agents SDK
-> (see [#71](https://github.com/LLMQuant/quant-mind/issues/71)). PR5 lands the
-> apex layer (`flows/` + `magic.py`); the remaining work is the `mind/`
-> memory + store layer scheduled for PR6 and PR7.
+Paper Flow V1 is source-first: code preserves the exact PDF revision and
+page-aware chunks before accepting a bounded, cited model summary. See the
+[complete persist/reopen/search example](examples/flows/paper.py) and
+[design contract](contexts/design/flow/paper.md).
 
 ---
 
@@ -238,7 +239,7 @@ asyncio.run(main())
 - [x] Better `flow` design for user-friendly usage
 - [x] First production level example (Quant Paper Agent)
 - [ ] Migrate Agent layer to OpenAI Agents SDK
-- [ ] Standardize knowledge format with `knowledge/` (Pydantic-based)
+- [x] Standardize knowledge format with `knowledge/` (Pydantic-based)
 - [ ] Additional content sources (financial news, blogs, reports)
 - [ ] Cross-step working memory (`mind/memory`) for batch document processing
 
@@ -254,18 +255,10 @@ QuantMind is designed with a larger vision: to become a comprehensive intelligen
 The foundation we're building today—starting with papers—will expand to encompass the entire financial information ecosystem.
 
 > [!NOTE]
-> **Future Conceptual Example (PR6 brings `FilesystemMemory`):**
->
-> ```python
-> from quantmind.configs.paper import ArxivIdentifier
-> from quantmind.flows import paper_flow
-> from quantmind.knowledge import Paper
-> from quantmind.mind.memory import FilesystemMemory  # PR6
->
-> memory = FilesystemMemory("./mem/factor-research/")
-> for arxiv_id in arxiv_ids:
->     paper: Paper = await paper_flow(ArxivIdentifier(id=arxiv_id), memory=memory)
-> ```
+> The current source-first paper path produces independently versioned source,
+> chunk-set, and cited-summary artifacts. Future agent memory and cross-document
+> reasoning can build on `LocalKnowledgeLibrary.search()` without changing
+> those canonical artifacts.
 
 This future state represents our commitment to moving beyond simple data aggregation and toward genuine machine intelligence in the financial domain.
 

contexts/design/README.md

@@ -24,12 +24,14 @@ implementation must preserve.
 
 | Domain | Design |
 |---|---|
-| Flow | [Paper extraction from input to validated result](flow/paper.md) |
+| Flow | [Source-first paper flow](flow/paper.md) |
+| Knowledge | [Paper sources, artifacts, citations, and locators](knowledge/paper.md) |
 | Flow | [News collection](flow/news.md) |
 | Preprocess | [Page-aware multimodal PDF parsing](preprocess/pdf.md) |
 | RAG | [Page-aware document chunking and retrieval](rag/document.md) |
 | Library | [Local knowledge storage and meaning-based search](library/local.md) |
 | Operations | [Public operation naming](operations/naming.md) |
+| Operations | [Orchestration and construction altitude](operations/orchestration.md) |
 
 ## Organization Rules