feat(scholar): scholarly-graph federation (OpenAlex + Semantic Scholar + Crossref) with citation-edge provenance

[HarperZ9]

What this ships

gather.scholar, a scholarly-graph federation adapter that unifies OpenAlex + Semantic Scholar + Crossref into gather's intake, riding on the existing accountability layer rather than replacing it.

The feature-leadership property: a research-intake organ should reach the scholarly graphs best-in-class, and it does so here without loosening a single accountability claim. Every paper still carries a provenance receipt, every run still folds into a witnessed digest, and the new citation graph is sealed the same way.

Citation edges as first-class provenance

A reference or a citation is not a number on a record; it is an edge with an origin. citation_edges turns each from -> to link into its own re-checkable receipt (method="citation-edge", registered DIRECT on the method ladder), recording which provider asserted it. Edges dedup by (from, to, direction, provider) and fold into the same witnessed digest as the papers, so the seal covers the graph (nodes and links), not just the nodes. An edge witnesses a claim the provider made, never a fact gather verified, and the receipt says so.

Dedup by DOI, no provenance dropped

The same paper surfaces from all three graphs. federate joins works on their normalized DOI (normalize_doi strips https://doi.org/ / doi: prefixes and lowercases) into one unified record. The unified item is a DERIVED compiled item whose derived_from points back at every provider's contribution, the same discipline as the corpus store: bodies deduped, receipts never. A DOI is the only join key; a title or author overlap never merges two works, and a DOI-less work stays its own record.

Zero-dep / offline-testable posture kept

• Core stays standard library: json and urllib only, no new dependency.
• Pure per-provider parsers (parse_openalex, parse_semanticscholar, parse_crossref); OpenAlex abstracts are reconstructed from its inverted index.
• ScholarSource is the isolated impure edge behind the Source shape, with an injectable fetcher so the whole federation is driven offline over recorded fixtures in tests. No live network is touched by the suite.

CLI

gather scholar QUERY [--providers openalex,semanticscholar,crossref] [--no-federate] [--edges] [--json]

--edges folds citation edges into the witnessed digest as first-class receipts.

Honest reach labels

• An abstract is what these APIs return, not the full paper. Crossref usually carries only metadata, so a Crossref work's text is its title/metadata, labelled by its crossref-api method, never dressed up as the paper.
• OpenAlex and Crossref list responses carry no incoming citers, so citations stays honestly empty for those providers; Semantic Scholar carries both directions.
• The unified record's abstract is the first provider that supplied one, and abstract_provider records which graph it came from.

What is shipped vs specced

Fully shipped and tested, end to end: the three parsers, DOI normalization + federation, citation-edge receipts, the ScholarSource edge, and the CLI scholar command (verified running the full CLI path offline via a stubbed network edge). Nothing is left as a spec stub.

Test evidence

New suite tests/test_scholar.py (33 tests over recorded fixtures, one paper described in all three graphs' native shapes):

$ python -m pytest tests/test_scholar.py -q
.................................                                        [100%]
33 passed in 0.07s

Full suite green (33 new + 289 pre-existing):

$ python -m pytest -q
........................................................................ [ 22%]
........................................................................ [ 44%]
........................................................................ [ 67%]
........................................................................ [ 89%]
..................................                                       [100%]
322 passed in 4.60s

CLI path exercised offline (stubbed net.http_get):

gathered 1 paper(s), 6 citation edge(s)
  paper          10.1234/monotile             An Aperiodic Monotile
digest seal: 217d6bfd34f02d92... | verified: True | covers papers + edges
exit 0

Lint/type on changed files: ruff check passes; mypy src/gather/scholar.py reports 0 errors (the 5 remaining package-level mypy errors are pre-existing in browser_evidence.py from PR #6, untouched here).

Limitations (honest)

• Fixtures are hand-authored to match each API's documented response shape, not captured from a live call; a schema drift at a provider would need a fixture refresh. The parsers degrade gracefully (skip malformed records, honest empty fields) rather than raising on unexpected shapes.
• Providers are queried sequentially in the impure edge; no rate-limit/backoff policy is wired in yet (the existing gather.federation_policy retry/backoff surface would be the natural composition seam for that, left as a follow-up).
• The live edge does DOI-vs-search URL routing per provider; pagination beyond the first page is not implemented.

Docs

README module list, CLI examples, and roadmap updated; CHANGELOG Unreleased entry added.

🤖 Generated with Claude Code