Hardens the P1 foundation and adds the derive seam that makes synthesis honesty
mechanical rather than documented.

Provenance hardening (P1 review):
- video: de-ladder real YouTube auto-caption rolling-window VTT (prefix-merge),
  handle NOTE blocks and HTML entities, stamp auto-captions method="auto-caption"
  (manual-first fallback), raise ValueError on malformed yt-dlp JSON instead of
  swallowing, log caption failures to stderr instead of silent absence.
- digest: the seal now folds in source/ref/method/derived_from, not just the content
  hash, so relabelling how an item was obtained breaks the seal as content tampering does.
- item: declare Provenance.derived_from (content hashes of an inference's inputs).
- .gitignore: .env / .env.* / CLAUDE.local.md.

The derive seam (gather.derive):
- derive(): pure builder for a derived item; sha256 over the inference, derived_from
  set to the inputs' content hashes (re-checkable pointers). Refuses empty inputs.
- Synthesizer protocol + NullSynthesizer: the optional model edge. The Null default
  performs a deterministic, extractive compilation (method="compiled") and invents
  nothing; a real abstractive synthesis (method="synthesized") only happens when a
  model is plugged into the seam. Gather cannot dress up a synthesis it did not perform.
- synthesize_item(): stamps the result with the synthesizer's own method.

26 tests (was 15), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ral)

Addresses the whole-branch review before release.

Critical:
- C1: derive() defaulted method="synthesized" and was bypassable, making the README's
  headline honesty claim false. Now derive() defaults to "compiled" and refuses to stamp
  "synthesized" at all; that label is producible only through synthesize_item with a model
  plugged into the Synthesizer seam (a shared internal _make_derived builder backs both).
  The guarantee is now structural, not a naming convention.

Important:
- I2: the digest seal now folds in title (a displayed, load-bearing field), so retitling a
  receipt breaks the seal; Digest docstring states what is sealed and that meta is not.
- I3/I4: gate the prefix-growth collapse to auto-captions only (transcript_from_vtt(auto=)),
  so manual subtitles pass through untouched and a manual cue that begins with the previous
  line is never silently merged away. parse_video gains auto_captions (drives both the
  collapse and the auto-caption label); CLI parse gains --auto-captions. Docstring now states
  honestly what it collapses and that head-dropping sliding windows are not stitched.

Minor:
- M5: derived_from is kept in original order in the seal, so reordering an inference's inputs
  is detected.
- M6: a whitespace-only   cue is dropped (tested).
- M7: demo smoke test pins the load-bearing facts (3 valid receipts, digest verifies, tamper
  caught), immune to hash churn; README seal value refreshed.
- M8: transcript docstring narrowed to yt-dlp-style WebVTT.

32 tests (was 26), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Hardens P1 and adds gather.derive: a structural synthesis-honesty guarantee
(synthesized only through the Synthesizer seam), the seal folds in title and
derived_from, and auto-caption de-laddering is gated so manual subs pass clean.
32 tests, ruff + mypy clean, zero-dependency core.

…docs)

Broadens reach while keeping the core zero-dependency and the network isolated.

- gather.net: the single network primitive. http_get (urllib, http/https only,
  capped body, Gather UA) and a pure, tested decode_body (charset from Content-Type,
  utf-8 fallback, never raises). HTTP lives here and in adapter fetches, nowhere else.
- gather.web: static web pages. A pure html.parser-based extractor (drops script/style,
  block elements become line breaks, entities decoded) yields (title, text). Honest about
  reach: no JavaScript is run, so a client-rendered page yields its shell and the receipt's
  http-get method says so. parse_web is pure and tested; WebSource is the impure edge.
- gather.feed: RSS and Atom. One pure parser handles both by local tag name (namespaced
  Atom and bare RSS), prefers the Atom alternate link, raises ValueError on malformed XML.
- gather.docs: local text files or a directory walk (sorted, deterministic, recursive,
  by extension), UTF-8 with replacement. The impure edge is the filesystem, not the network.
- cli: new web/feed/docs commands, all taking the shared --scope/--json; DRYed via helpers.

46 tests (was 32), ruff + mypy clean. Pure parsers tested without network; live fetch is
the thin isolated edge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ectness)

Addresses the whole-branch review before release.

Critical:
- C1/C4: real SSRF defense. http_get now enforces the scheme allowlist AND a
  private/loopback/link-local/reserved host block on the initial URL and on every redirect
  hop (a custom redirect handler), so a public URL that redirects to cloud-metadata or an
  internal host is refused. The previous scheme-only check protected nothing once a redirect
  was in play. _host_is_private fails closed on unresolvable hosts. Residual (DNS rebinding)
  is documented, not hidden.
- C2: decode_body now unquotes the charset token, so a legal charset="latin-1" (quoted) is
  honored instead of silently mangled to utf-8.
- C3: feed entry selection no longer walks the whole tree (root.iter); it selects item/entry
  that are direct children of the root or a channel, covering RSS 2.0, RSS 1.0/RDF (items at
  root), and Atom, without mistaking an entry nested in an Atom <source> for a post.

Important:
- I7: feeds are parsed from bytes, not the decoded str, so a feed carrying an XML encoding
  declaration (most real feeds) parses correctly instead of raising. ElementTree honors the
  declared encoding.
- I2: table cells, dd/dt, and more sectioning tags break, so HTML cells no longer run together.
- I4: Atom content/summary declaring HTML is run through the text extractor; an untyped RSS
  description is preserved verbatim (stripping untyped text would risk mangling plain content).
- I5: docs records each file's resolved real path in ref, so a symlinked source is attributed
  to where it actually lives; symlinked dirs are not descended (no walk loops).
- I3/M5: docstrings state the convert_charrefs guarantee (output is final text, not markup)
  and that a binary file read as text yields a faithful receipt of an imperfect reading.

Minor:
- M1: max_bytes threaded through WebSource/FeedSource. M2: truncation warns on stderr.
  M3: link precedence made explicit (alternate, then text link, then any href).

Adversarial tests added for every fix (quoted charset, private-host block, nested entry,
RDF-at-root, bytes-with-encoding, Atom html strip, table cells, unclosed skip tag).
55 tests (was 46), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds web (static http), feed (RSS/Atom), and docs (local files) adapters plus a
single isolated network primitive (gather.net) with a real SSRF guard (scheme +
private-host block on every redirect hop). Feeds parse from bytes to honor encoding
declarations. 55 tests, ruff + mypy clean, zero-dependency core.

Adds the papers lane behind the same Source seam.

- gather.arxiv: the arXiv API returns Atom, so parse_arxiv is a pure, tested parser. Each
  paper Item carries the ABSTRACT (the API returns abstracts, not full text) plus authors,
  categories, primary category, date, DOI, and the PDF link in meta. The method "arxiv-api"
  and the abstract-as-text are honest: an abstract is never mistaken for the paper.
  arxiv_query_url distinguishes a bare arXiv id (fetch by id) from a free-text search.
- gather.pdf: local PDF text via the pdftotext CLI (poppler-utils), the isolated external-tool
  edge like video's yt-dlp. The receipt's "pdftotext" method records this is a tool's
  best-effort reading (scanned PDFs yield little, layout can reorder), not the authoritative
  document. Decodes pdftotext output as UTF-8 explicitly (avoids the Windows locale pitfall).
- cli: arxiv and pdf commands, sharing --scope/--json; arxiv takes --max-results.

60 tests (was 55), ruff + mypy clean. Core stays zero-dependency; pure parsers tested
without network or tool.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ifier tests)

Addresses the whole-branch review before release.

Important:
- pdf: insert "--" before the target in the pdftotext argv, so a file named like "-f"
  cannot be parsed as a flag (argument injection).
- arxiv: extract is_arxiv_id as the single named classifier and test it across new-style
  (4/5-digit, versioned), old-style (hep-th/9901001, math.CO/0601001) ids, and queries
  (including a 6-digit near-miss); the load-bearing regex was effectively untested.
- arxiv: the receipt now records HOW a paper was found, method "arxiv-api-id" for a direct
  id lookup vs "arxiv-api-search" for a relevance search, so a search hit is not recorded as
  a precise id fetch. An entry with no id is skipped (real arXiv always sends one).

Minor:
- net: User-Agent derives from __version__ (was a stale "gather/0.3"), so it never misreports.

73 tests (was 60), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds the papers lane: gather.arxiv (pure Atom parser over the arXiv API, abstract +
metadata per paper, id-vs-search recorded in the method) and gather.pdf (pdftotext
external-tool edge). 73 tests, ruff + mypy clean, zero-dependency core.

…iable store

Turns one-shot gathers into a corpus that stays provable over time.

- gather.store.Corpus: bodies stored content-addressed (objects/ab/cdef...), keyed by the
  sha256 of their content, so identical content is stored once (natural dedup); catalog.jsonl
  is an append log of one receipt row per distinct item. verify() re-hashes every stored body
  and reports MATCH / MISSING / CORRUPT, making the digest's proof-over-trust durable over a
  growing corpus. Bodies are written to a temp file then renamed (never half-present).
- Efficiency (a stated pain point): content-addressing dedups bodies; the catalog is read as a
  stream (rows(), one row at a time, never the whole corpus in memory); the corpus digest seals
  from the rows alone, without re-reading bodies.
- digest: extract digest_of_receipts so a digest can be folded from stored catalog rows.
- cli: every gather command gains --store DIR (persist what was gathered); new corpus command
  with list / verify / digest (verify exits non-zero if anything is MISSING or CORRUPT).

A stored corpus seals identically to the live items it came from. 79 tests (was 73),
ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…aversal guard, fsync)

Addresses the whole-branch review before release.

Critical:
- C2: dedup no longer drops provenance. It dedups BODIES only (content-addressed, stored
  once); every distinct receipt is kept, so two different items with identical text keep both
  receipts sharing one body. Re-adding an item whose whole receipt already exists is the no-op
  that is deduped. The headline "a receipt on every item" now holds for the corpus too.
- C1: with all distinct receipts kept, the corpus seal equals the live digest(items) even
  across multiple adds and shared bodies (test added). Documented the precise contract.
- C3: path-traversal closed. A content hash is validated as 64 hex chars before it is used to
  build any object path, so a tampered/hand-edited catalog sha cannot read outside the corpus.

Important:
- I1: real durability. Bodies and the catalog are fsync'd (and the dir on POSIX) by default;
  "durable" is now earned. fsync=False opts out for a faster bulk load (the efficiency trade,
  documented).
- I2: rows() raises a located ValueError on a malformed catalog line instead of an opaque
  crash; single-writer assumption documented.
- I3: the streaming claim is now accurate (bodies read one at a time, never all held; the small
  receipt rows are collected to seal).
- I4: digest_of_receipts validates fields and raises a clear error naming what is missing.
- I5: non-JSON-serializable meta raises a clear error naming the item; meta round-trip (and its
  JSON type limits) documented and tested.

Tests added: distinct-items-share-a-body keep both, identical-receipt deduped, meta round-trip,
traversal-guard, multi-add digest equivalence, missing-field rejection.
84 tests (was 79), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds gather.store.Corpus: bodies content-addressed and deduped, every distinct receipt
kept (no provenance dropped), verify() re-hashes every body (MATCH/MISSING/CORRUPT),
fsync durability, path-traversal guarded. Every command gains --store; new corpus
list/verify/digest. 84 tests, ruff + mypy clean, zero-dependency core.

Ties intake into one re-checkable session.

- gather.run.gather_run: orchestrates a session, fetch each (source, target) job, collect,
  scope-filter, optionally fold into one synthesized item through the Synthesizer seam, digest,
  and optionally persist to a corpus. The clock is injected, so given the same fetched items
  the run is deterministic and replayable.
- RunRecord: a witnessed, re-checkable record of the session (when, targets, scope, counts,
  whether a synthesis was added, the items' digest seal) with its own seal over those fields.
  verify_record recomputes it; altering a counted field breaks it.
- Composition seams (peer-composition): the scope filter and the synthesizer are pluggable and
  default to Null (keyword scope, no synthesis), so the run stands alone and a model-based scope
  or synthesizer plugs in without the run importing it.
- store: durable run history (add_record / runs); the JSONL streaming is shared with the catalog.
- cli: a run command driven by a JSON config (jobs, scope, store, synthesize) over a lazy source
  registry; corpus gains a runs action.

89 tests (was 84), ruff + mypy clean. Run tested deterministically with in-memory sources.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…, honest counts)

Addresses the whole-branch review before release.

Critical:
- C1: a persisted run record is now genuinely re-checkable. RunRecord.from_dict reconstructs
  a record from its stored form, and `corpus runs --verify` recomputes every record's seal from
  disk (exit non-zero if any is bad). Before, the record sealed in-process but nothing read it
  back, so the witness died at process exit.
- C2: a test pins the seal surviving a full JSON round-trip (dict -> json -> from_dict -> verify),
  the property the re-checkable seal depends on.
- C3: the run's digest_seal fingerprints THAT run's items; a corpus's own digest fingerprints the
  whole corpus, so they coincide only for the first run. Documented, and a second-run divergence
  test pins it (it was implied to be a general guarantee).

Important:
- I1: the run command's config load and job construction are now guarded, so a missing file,
  bad JSON, a job missing source/target, or an unknown source give a clean "run failed: bad
  config: ..." message instead of a raw stacktrace; network errors stay distinguishable.
- I4: RunRecord gains a digested count (kept plus the synthesis), so kept/synthesized/digested/
  stored reconcile instead of an unexplained off-by-one. Tested.
- I2/I3: documented the two-phase persistence (items then record, so a crash can only lose the
  witness, never the items) and that the synthesized item is admitted on its in-scope inputs'
  provenance, not re-scoped on its own text; a model-synthesizer-out-of-scope test pins it.
- store seam typed (StoreLike Protocol), matching the Source/Synthesizer seam discipline.

94 tests (was 89), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds gather.run: orchestrates fetch/scope/synthesis/digest/store into one re-checkable
RunRecord (re-checkable from disk via from_dict + corpus runs --verify), with pluggable
scope and synthesizer seams defaulting to Null. Durable run history in the corpus.
94 tests, ruff + mypy clean, zero-dependency core.

The efferent bridge: downstream organs draw scoped, re-verifiable subsets from the corpus.

- gather.recall.Query + recall(): query a stored corpus by scope terms (matched against title
  and body) and exact-match source/kind/method filters (all ANDed). Streams the catalog: the
  cheap metadata filters are answered from the row before any body is read, only survivors have
  their body loaded to check terms and be returned; order-preserving with an early-stop limit.
  Returned items are reconstructed, so they re-verify against their receipts.
- cli: corpus gains a search action (--terms/--source/--kind/--method/--limit) that prints the
  matches and a scoped digest of them; --json emits the recalled catalog and digest.

100 tests (was 94), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tems)

Addresses the whole-branch review before release.

Important:
- I2: recall no longer crashes on a missing body or silently returns a corrupt one. It loads,
  applies the term filter, then re-verifies each item to be returned; a row whose body is gone is
  skipped as MISSING and one whose body no longer hashes to its receipt as CORRUPT.
  recall_audited returns (items, skipped) so corruption is surfaced; recall returns just the
  verified items. The CLI search reports "N match(es), bodies verified, M skipped" and exits
  non-zero if anything was skipped.
- I1: limit <= 0 now returns nothing (was an off-by-one returning one arbitrary item).
- I3: the search output no longer prints a meaningless "verified True" for an in-process digest;
  it says "bodies verified" (true, since recall re-verifies) and special-cases zero matches.
- I4 / M1: documented that within a filter values are OR and across filters AND, and that terms
  are case-insensitive substrings (so 'art' matches 'cartesian'), in the Query docstring, CLI
  help, and README.

Tests added: limit 0/negative, missing-body skip, corrupt-body skip, OR-within-a-filter,
substring match, title-only match.
106 tests (was 100), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds gather.recall: Query a stored corpus by substring scope terms and source/kind/method
filters (OR within, AND across), returning reconstructed items that are re-verified (missing
or corrupt bodies skipped and reported). New corpus search CLI action. 106 tests, clean,
zero-dependency core.

… method ladder

The hard-source honesty layer.

- gather.credentials: the one place secrets enter. require_secret(name) reads from the
  environment, raises naming only the missing variable (never a value), and the value is never
  logged, never stored in an Item or receipt, never put in a URL.
- gather.api: an authenticated JSON-API adapter, the worked example of the pattern. The token is
  read from env via require_secret and sent in an Authorization header (so it never reaches the
  URL, the receipt, or disk); parse_api turns the response into one Item per record, pure and
  tested. net.http_get gained an optional headers param (sent, never logged).
- gather.method: the method ladder. directness() classifies a method as direct (a fetch/read/
  transcription) or derived (compiled/synthesized); is_consistent() checks an item's method
  agrees with its derivation chain (a derived item carries derived_from, a direct one does not),
  making the "a quote is never an inference" rule checkable, not just documented.
- cli: an api command and registry entry.

119 tests (was 106), ruff + mypy clean. Core stays zero-dependency; secrets tested via
monkeypatched env, the API parser tested without network.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…forced)

Addresses the whole-branch review before release. The credential surface was found sound;
the blockers were both in the method ladder.

Critical:
- C1: VideoSource stamped the configured yt-dlp binary path as the method, so a non-default
  path made every video item classify as "unknown" on the ladder. It now emits the stable
  constant "yt-dlp".
- C2: the method ladder was dead code while the README implied it was active. make_item now
  ENFORCES it: a direct method (a fetch/read) cannot carry derived_from and a derived method
  (compiled/synthesized) cannot lack it; an unregistered method makes no claim. The honesty is
  mechanical, like derive() refusing to stamp "synthesized". (method.py restructured to avoid an
  import cycle: it carries the Item type only under TYPE_CHECKING.)

Important:
- I4: a test now proves the headline guarantee, that ApiSource sends the token in the header and
  it appears in no item, no receipt, and neither stdout nor stderr.
- I1: a non-string text_key field is serialized as canonical JSON, not a Python repr, so the
  receipt always fingerprints JSON.
- I2: ApiSource refuses a token that appears in the URL (ref is witnessed; secrets ride the header).
- I3: require_secret rejects a whitespace-only or newline-bearing secret (a stray line ending
  would corrupt or inject a header).
- I5: the api CLI command now honors --id-key/--title-key, matching the run config path.

122 tests (was 119), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ed method ladder

Adds gather.credentials (env-only secrets, never witnessed), gather.api (authed JSON-API
example, token in header), and gather.method (the direct/derived ladder, enforced in
make_item). 122 tests, ruff + mypy clean, zero-dependency core.

…split)

- ARCHITECTURE.md: the design map (the Source seam, the receipt and method ladder, the derive
  seam, the network edge, credentials, the corpus, the witnessed run, recall, determinism, the
  threat model, peer-composition).
- CHANGELOG.md: the version history, 0.1 through 0.9.
- examples/pipeline.py: the whole organ in one offline run (run -> store -> verify -> recall,
  with the record re-checking from disk), deterministic, no network.
- Split the CLI: cli.py is now the argument surface, commands.py the implementations, so no module
  exceeds the size budget (cli 115, commands 258).
- README links the new docs and the second example.

122 tests, ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ARCHITECTURE.md, CHANGELOG.md, examples/pipeline.py (the whole organ offline), and the
cli/commands split. 122 tests, ruff + mypy clean, zero-dependency core.

…xes)

A three-lens whole-system review (correctness, security, docs) gated the milestone. Its
findings are fixed here.

Security:
- Strip credentials (Authorization/Cookie/Proxy-Authorization) on a cross-origin redirect, so a
  compromised or open-redirecting endpoint cannot harvest a bearer token (CVE-2018-18074 class).
- Refuse caller-supplied routing headers (Host/Forwarded/X-Forwarded-*) that could desync the
  URL-based host guard.
- Harden the persisted run history: RunRecord.from_dict raises a clear error on a malformed row
  and corpus runs --verify marks it bad instead of crashing.

Integrity (the seal is the product):
- The digest seal canonicalizes each receipt as a named-key JSON object, not a positional array,
  so the receipt-to-bytes mapping is unambiguous by construction and no field-value permutation
  can collide.
- gather_run dedups duplicate receipts before sealing, so a run's digest seal equals the seal of
  what the corpus stored (a source listing the same item twice no longer diverges).
- recall and verify now agree a non-hex/tampered sha is CORRUPT, not MISSING.

Robustness:
- gather run rejects a non-list scope (a bare string would become char-terms and be sealed).
- feed entries with no derivable identity are skipped, matching arxiv.
- _object_path binds the validated sha once; Corpus.add documents its per-call dedup cost.

Public API + docs:
- A curated top-level surface is re-exported from gather with __all__.
- Stale P1/P2 docstrings removed; README adds gather.commands; "HTTP transport" wording; the
  substring-match and single-writer limits surfaced; the DNS-rebinding residual and the
  Windows fsync caveat documented; the LICENSE security-contact URL corrected to gather.

132 tests (was 122), ruff + mypy clean, zero-dependency core. ARCHITECTURE.md, CHANGELOG.md,
and examples/pipeline.py complete the release.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The accountable research-intake organ, stable. Eight adapters behind one Source seam, a
provenance receipt on every item, the enforced method ladder, the synthesis derive seam,
a content-addressed re-verifiable corpus, the witnessed gather run, recall, isolated
credentials, and the digest seal as the re-checkable witness.

Gated by a three-lens whole-system review (correctness, security, docs). 132 tests,
ruff + mypy clean, zero-dependency core, fair-source.

…e seam

The "reach anywhere" promise delivered, each as an isolated external-tool edge like video/pdf.

- gather.browser: JavaScript-rendered pages via a headless Chromium (--dump-dom), then the pure
  web text extractor. The receipt's browser-extract method records that JavaScript was run (vs the
  static web adapter's http-get), and the same scheme + private-host SSRF guard is applied to the
  target via the shared net.validate_public_http_url.
- gather.ocr: text from a scanned image via tesseract; the ocr method records a machine reading.
- gather.transcribe: a transcript from audio via a Whisper-style CLI (temp-dir output, read back
  like captions); the transcribe method records a machine transcription.
- External-tool paths are resolved to absolute paths so a filename starting with - cannot be read
  as a flag; net.validate_public_http_url extracted and shared by the http and browser edges.
- cli: browser/ocr/transcribe commands + registry entries.

138 tests (was 132), ruff + mypy clean. Pure parsers/builders and guards tested; the live tool
call is the isolated edge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… opt-in)

The review found accountability defects (this project's first-class concern), not exploits.

- C2: the Chromium sandbox is left ON by default; --no-sandbox is now opt-in (constructor flag +
  CLI flag), documented as a deliberate hardening downgrade for running as root in a container.
- C1: stop claiming the browser has the http edge's SSRF guard. The class docstring, ARCHITECTURE
  threat model, and CHANGELOG now state plainly that the guard covers only the initial navigation;
  a rendered browser then follows its own redirects and sub-requests unguarded (a wider DNS-rebind
  window), so it must not run against untrusted URLs where internal services are reachable.
- C3: the README intro no longer says the hard sources are unshipped (it contradicted the rest of
  the file); browser/ocr/transcribe are added to the ARCHITECTURE adapter list with the browser
  caveat where a user reads "JavaScript-walled pages".
- I2: the browser refusal test also covers data:/javascript: schemes.
- I3: a test pins that the guard raises BEFORE any subprocess spawns (order is the security property).
- M1: http_get uses validate_public_http_url's returned (trimmed) value.

139 tests (was 138), ruff + mypy clean. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Adds gather.browser (JS pages via headless Chromium), gather.ocr (tesseract), and
gather.transcribe (Whisper-style), each an isolated external-tool edge. The browser edge
is documented honestly: initial-navigation guard only, sandbox on by default. 139 tests,
ruff + mypy clean, zero-dependency core.

The Synthesizer seam has shipped since 0.2 but only the Null (compile) implementation existed.

- gather.model.SubprocessSynthesizer: the real model edge. Shells to an operator-configured model
  CLI to infer a statement, with the prompt (built from gathered, possibly untrusted content)
  written to STDIN, never the argv, so no gathered text can be parsed as a flag. method
  "synthesized", so synthesize_item stamps it a genuine inference with derived_from set and the
  digest seal records it honestly. build_prompt is pure (bounds each input).
- gather run accepts a "synthesizer" command list in its config to use the model edge; otherwise
  "synthesize": true uses the compiling Null.

144 tests (was 139), ruff + mypy clean. The edge is tested with a real subprocess (the test
interpreter), not a mock. Core stays zero-dependency.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fetch.py wraps net.py's network edge without weakening it (same scheme
allowlist, SSRF host block, and cross-origin credential stripping on every
redirect hop) and adds what a scraper needs but none of the competitors
witness:

- FetchReceipt: sha256 of the exact bytes, a response-headers digest, the
  recorded redirect chain, status, and attempt count; verify(body) re-checks it.
- conditional GET (ETag / If-Modified-Since) with honest 304 handling.
- retry with exponential backoff on transient failures.

The transport is a seam, so retry/conditional/receipt logic is tested offline
with a fake transport. Default UA identifies gather rather than impersonating a
browser, and any caller-supplied stealth headers stay on the record.

7 new tests (retry, exhaustion, tamper, routing-header guard); suite 312 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Capability first: crawl.py is a real crawler - concurrent wave fetching,
BFS/DFS frontier, URL canonicalization + dedup, robots.txt (stdlib
robotparser), sitemap discovery, depth/page caps, per-host throttle, and
pause/resume via a serializable CrawlState. That is crawlee/firecrawl-class
crawl orchestration.

Then the multiplier: an append-only, hash-chained CrawlLedger. Each record
links to the previous by hash, so a reviewer can re-derive the chain and prove
the crawl was not reordered, truncated, or edited after the fact.

This lands the strategy correction: the receipt is a multiplier on top of
best-in-class capability, never a substitute for it. Goal doc rewritten to a
capability-superset-first frame with an optional-backend parity layer.

Fetcher is a seam; all orchestration tested offline. 11 new tests (dedup,
depth/page caps, robots, sitemap, resume chain continuity, concurrent-worker
parity, ledger tamper detection); full suite 323 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Two capabilities, one accountable:

- extract_schema: a schema of CSS-lite selectors (+ optional attr/regex/many)
  to a JSON record, every field bound to its source node path and content hash.
  The firecrawl `extract` shape, made deterministic and re-verifiable.
- verify_record: given a record proposed by anything (an LLM, an API, a human),
  confirm each value is GROUNDED in the fetched content; reject any value that
  appears nowhere on the page. Turns LLM extraction from trust-the-model into
  prove-it-against-the-source, which none of the competitors do.

6 new tests including tamper detection and a hallucinated-field rejection;
full suite 329 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…ore)

The parity layer's accountable half, zero-dep. backends.py is a capability
registry: backends declare what they can do (js-render, stealth, fast-parse),
render() resolves the best available, and a required-but-missing capability
returns UNVERIFIABLE with a reason - never a static shell dressed up as a
render. Every result records which backend produced it.

best_parser/detect_fast_parse pick a native parser (lxml/selectolax) when
installed, else stdlib. The heavy backends register themselves when their
optional dependency is present; none is bundled.

7 new tests including the honest-degrade negative; full suite 336 passed.

The actual browser/stealth/fast-parse backends (wedge 5-backends) are gated on
an operator decision about optional dependencies plus an environment where they
can be installed and verified; documented in WEB-ENGINE-UPLIFT.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…edge 7 partial)

DX + performance, all unblocked and zero-dep:

- cache.py: dev-mode response cache (Scrapling dev-mode parity). Content-
  addressed store, offline replay (iterate your parser without re-scraping),
  and conditional revalidation that serves the cached body on a 304.
- export.py: uniform JSON / JSONL export across every receipt type.
- fetch.py: FetchReceipt now carries etag / last_modified (feeds cache
  revalidation).
- Performance: node paths are assigned once at parse time instead of rebuilt
  per call. extract over ~15k elements dropped from 1171 ms to 113 ms (~10x).
  examples/bench.py publishes the honest numbers.

9 new tests (cache tamper + replay-without-refetch, export roundtrips); full
suite 345 passed. Honest benchmark + competitor comparison in
docs/WEB-ENGINE-UPLIFT.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…5-backends)

The parity layer, as opt-in extras; core stays zero-dep. Missing edge degrades
to honest UNVERIFIABLE or stdlib fallback, never a fake.

- fastparse.py: lxml parser producing the IDENTICAL gather Node tree (paths
  match stdlib); verified ~2x faster (33 ms vs 69 ms on the ~15k-element bench).
- backends_browser.py: Playwright js-render backend. Verified it really launches
  headless Chromium and executes JavaScript. No browser binary -> UNVERIFIABLE
  with reason, via render()'s new failure-degrade path.
- backends_stealth.py: curl_cffi TLS-impersonation transport for the accountable
  fetch path (same FetchReceipt), SSRF guard re-applied per redirect hop and
  cross-origin credentials stripped.
- pyproject extras: fast / browser / stealth / all.
- default_registry now honestly reports installed capabilities.

18 new tests (gating, render-failure degrade, stealth redirect + credential
strip + transient 5xx); full suite 356 passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…wedges 6, 7)

- search.py: pluggable search-provider seam returning a SearchReceipt of
  SOURCE_LEADs (never verified content); honest UNVERIFIABLE with no provider;
  search_and_fetch chains leads into the accountable fetch path; searx_provider
  is a no-API-key SearXNG backend. This is firecrawl search/agent parity.
- CLI: `gather caps` (honest capability report), `gather extract` and
  `gather markdown` (URL or local file), `gather crawl` (witnessed ledger JSON).
  Every command prints a receipt.

8 new tests; full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

interop.py maps gather's web-data receipts (extraction, fetch, crawl ledger,
search leads) into organ-bundle interchange entries, so a gather crawl can feed
index's context, a gather extraction can back a forum evidence lane, and a
gather provenance receipt can seed a learn lesson - all through the one shared
spine that index/forum/learn/crucible/emet already compose on.

Entries ride the registered gather-corpus kind (subtype in the summary), so they
validate against proof-surface's real organ_receipt_bundle validator TODAY
(verified: 0 issues) without any cross-repo change. crawl_entry reflects ledger
integrity (tampered chain -> failed entry, not a silent pass).

4 new tests; full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

- README: new "The web-data engine" section positioning gather's fetch/extract/
  track/crawl/schema/search + capability backends against the four competitors,
  in-voice; "organic completion at 1.5.0" narrative updated for the 1.6.0 front.
- USAGE: web-data commands (caps/extract/markdown/crawl) + optional-extra installs.
- CHANGELOG: 1.6.0 web-data engine entry.
- Version bumped 1.5.0 -> 1.6.0 (pyproject + __init__).
- ruff clean; mypy clean across the 15 new modules (import order, 4 real type
  fixes: cache fetch_fn narrowing + body reuse, crawl batch annotation,
  browser wait_until, caps dict indexing).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

An adversarial review (5 reviewers x module groups, each finding independently
verified) confirmed 10 real defects; all fixed with regression tests. The
verify pass correctly rejected 5 false alarms (relocation-tie determinism,
intended empty-skip, GONE evidence, a hash-chain misread, a retry off-by-one).

- fastparse: lxml wraps a fragment in synthetic html/body, so a fragment's node
  paths diverged from stdlib and track.relocate returned a FALSE RELOCATED where
  it should be MATCH. Unwrap the synthetic wrapper; paths now match either parser.
- schema_extract.verify: a tampered field value that is a substring of the source
  passed verification. Bind each value with its own hash (Hit.value_sha256).
- schema_extract.verify_record: substring grounding falsely accepted '100' inside
  '1000'. Now word-boundary grounding; fixed _haystack gluing adjacent inline
  element text (a latent bug the boundary change surfaced).
- cache: a 304 with no cache entry to revalidate poisoned the store with empty
  bytes. Now refused.
- backends_stealth: credentials were not stripped on an https->http downgrade
  redirect (CVE-2018-18074 class). Now track scheme and strip, mirroring net.py.
- search.urls / interop.validate / export.to_dict/to_json: crashes on partial
  hits, malformed bundles, reference cycles, and non-serializable leaves. Now
  defensive (report/serialize, never raise/overflow).

11 regression tests added; full suite 379 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…r + Crossref) with citation-edge provenance (#8)

* feat(scholar): federate OpenAlex + Semantic Scholar + Crossref with citation-edge provenance

Add gather.scholar, a scholarly-graph federation adapter unifying OpenAlex,
Semantic Scholar, and Crossref into one intake. Citation edges (references and
citations) become first-class provenance receipts (method citation-edge,
registered DIRECT on the ladder) recording which provider asserted each link,
sealed into the digest alongside the papers. Records dedup by normalized DOI
into one unified compiled item whose derived_from points back at every
provider's contribution, so no provenance is dropped; a DOI is the only join
key, never a fuzzy title match.

Pure per-provider parsers, one isolated impure edge with an injectable fetcher,
so the whole federation is tested offline over recorded fixtures. CLI: gather
scholar QUERY [--providers ...] [--no-federate] [--edges] [--json].

33 new tests over fixtures; full suite 322 passing; scholar.py ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: sort imports in tests/test_browser_evidence.py (ruff I001)

Fixes the CI lint gate on this branch; the import fix already landed on
main but this branch predated it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(browser-evidence): narrow after/verification to Mapping for mypy

Binding packet.get(...) once (instead of calling it twice across the
isinstance guard) lets mypy narrow the value to Mapping, clearing the
union-attr / arg-type errors on the CI type-check gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Zain Dana Harper <zain@aurora-framework.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Integrates the concept from Phil Holden's partialupdate (talk: "What if AI
replies in HTML not Markdown?"). The repo is gone (deleted from GitHub, not on
npm, no Wayback snapshot), so this rebuilds the idea from the talk and his
reactive/CRDT ecosystem, and adds gather's accountability layer.

stream.py parses an HTML stream incrementally (StreamingExtractor.feed(chunk))
and commits only STABLE blocks as their end tag arrives, holding back the
incomplete tail. Each stable increment is content-hashed and linked into an
append-only hash chain (StreamLedger), so a streamed extraction is a
re-verifiable ledger of stable increments; the open tail is explicitly PENDING
(pending()), never witnessed as final; an unclosed block at stream end is left
uncommitted (honest about truncation). Chunk boundaries do not change the
ledger (html.parser buffers incomplete tags), so feeding any split yields the
same commits, hashes, and order as feeding the whole document.

This is extract.py's stable-vs-incomplete honesty applied to a live stream, and
serves the talk's case directly: an AI streaming HTML, extracted and receipted
as it arrives. 5 tests (progressive commit, chunk determinism, tamper
detection, truncation honesty); full suite 384 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…act-track

# Conflicts:
#	CHANGELOG.md

The merge was green on 'ruff check src/gather' but CI runs ruff over src, tests,
and examples; test_interop imported SearchHit unused. Full CI gate verified
locally: ruff clean (all dirs), mypy clean (55 files), 417 passed at 85.69%
coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Part of the Project Telos visual identity refresh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Style: spectrum banner + feature-first README header

Patch release: visual-identity refresh (spectrum banner, feature-first README) plus docs overhaul (docs/INTRODUCTION.md, delivery contract) and a live PyPI downloads badge; no engine code changes since v1.6.0. Full suite: 417 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>