feat: premium-grade extraction & wikification engine

README.md

@@ -19,16 +19,26 @@ uv run wikifi init
 
 - `init` — one-time setup; scaffolds the `.wikifi/` directory and any local config the implementor chooses to expose.
 - `walk` — main entry point. Walks the target codebase and produces the wiki content.
+  - `--no-cache` — force a clean re-walk; drops the on-disk extraction + aggregation caches.
+  - `--review` — run the critic + reviser loop on derivative sections (personas, user stories, diagrams).
+  - `--provider {ollama|anthropic}` — override the configured provider for this walk.
+- `report` — print a coverage + quality report (per-section file counts, findings, body sizes).
+  - `--score` — additionally run the critic on every populated section for a 0-10 quality score.
 - `ask` — natural language queries against the wiki content, with optional context injection from the target codebase.
 - `chat` — interactive REPL for iterative exploration of the wiki content and the target codebase.
 
 ## Architecture
 - **`wikifi/` package** — the library, with the CLI entry point exposed via `[project.scripts] wikifi = "wikifi.cli:main"` in `pyproject.toml`.
 - **Repository introspection** — before walking, the agent reviews the target's root structure (manifests, top-level layout, gitignore signals) and decides which paths carry production source worth analyzing. The walk that follows is deterministic — the agent does not re-pick scope mid-walk.
-- **Per-file extraction** — for each in-scope file, the agent extracts contributions to each *primary* capture section (see `VISION.md`) into structured findings.
+- **Repo graph** (`wikifi/repograph.py`) — a regex-driven static analysis builds an import / reference graph across in-scope files, plus classifies each file's `FileKind` (application code, SQL, OpenAPI, Protobuf, GraphQL, migration, other). Each file's neighborhood is injected into the extraction prompt so per-file findings can describe cross-file flows.
+- **Specialized extractors** (`wikifi/specialized/`) — schema files (SQL, OpenAPI, Protobuf, GraphQL, migrations) bypass the LLM entirely and run through deterministic parsers. The structured findings reach the same notes store as LLM output, so the rest of the pipeline is unchanged.
+- **Per-file extraction** — for each in-scope file, the agent extracts contributions to each *primary* capture section (see `VISION.md`) into structured findings. Each finding carries a structured `SourceRef` (file + line range + content fingerprint) for downstream citation.
+- **Content-addressed cache** (`wikifi/cache.py`) — extraction findings are keyed by `(rel_path, sha256(file_bytes))`; aggregation bodies are keyed by a hash of the section's notes payload. Re-walks skip every file whose fingerprint hasn't changed; resumability after a crash is a free property of the same cache. Use `walk --no-cache` to force a clean re-walk.
 - **Input filtering** — the walker recognizes and skips unstructured or near-empty files (stub `__init__` files, empty fixtures, machine-generated artifacts) before they reach the agent. Empty input must never stall the walk.
-- **Section synthesis** — primary capture sections are synthesized from the accumulated per-file findings; derivative sections (personas, user stories, diagrams) are produced *after* primary content is complete, taking the synthesized primary content as their input.
-- **Provider abstraction** — the LLM backend is reached through a provider interface. Default is a local Ollama server; alternative providers (hosted Anthropic, hosted OpenAI, custom) plug in by implementing the same interface.
+- **Section synthesis** — primary capture sections are synthesized from the accumulated per-file findings; the aggregator emits a structured `EvidenceBundle` (body + claims + contradictions) and the renderer threads numbered citations + a "Conflicts in source" block into the section markdown. Derivative sections (personas, user stories, diagrams) are produced *after* primary content is complete, taking the synthesized primary content as their input.
+- **Critic + reviser** (`wikifi/critic.py`) — opt-in (`walk --review`), runs a quality pass on derivative sections: scores the body against its brief and upstream evidence, identifies unsupported claims, and re-synthesizes when the score is below threshold. Only accepts a revision if it scores at least as well as the original.
+- **Coverage + quality report** (`wikifi/report.py`) — `wikifi report` produces a per-section view of files contributing, finding count, body size, and (with `--score`) critic-derived quality scores.
+- **Provider abstraction** — the LLM backend is reached through a provider interface. Default is a local Ollama server (`OllamaProvider`); the hosted Anthropic backend (`AnthropicProvider`) is opt-in via `WIKIFI_PROVIDER=anthropic` and uses prompt caching with `cache_control: ephemeral` on the system prompt so the multi-KB extraction prompt is paid for once across hundreds of per-file calls.
 - **Wiki adapter** — writes the rendered wiki into the target's `.wikifi/` directory. Layout, taxonomy, and structure within `.wikifi/` are at the implementor's discretion, provided the content contract from `VISION.md` is met.
 
 ## Tech stack

TESTING-AND-DEMO.md

@@ -0,0 +1,251 @@
+# Testing & demoing the premium pipeline
+
+This document covers how to verify and demo the nine premium features
+landed in this PR. Every step works from a clean clone — no external
+service required for the test suite, and only Ollama (default) or an
+Anthropic API key (opt-in) for the live demos.
+
+## Prerequisites
+
+```bash
+make hooks                 # one-time, enables the pre-commit + pre-push hooks
+uv sync                    # installs anthropic + the other deps (already in uv.lock)
+```
+
+## Running the test suite
+
+```bash
+make test                  # runs pytest with coverage
+```
+
+Expectations:
+- **156 tests pass.**
+- **Total coverage ≥ 93%.** Every new module is at or above 86%; the
+  premium-pipeline modules — `fingerprint`, `cache`, `evidence`,
+  `critic`, `report`, `repograph`, `specialized/*`,
+  `providers/anthropic_provider` — each carry a dedicated test file.
+
+To run only the suites for the new functionality:
+
+```bash
+uv run pytest tests/test_fingerprint.py tests/test_cache.py tests/test_evidence.py \
+              tests/test_repograph.py tests/test_specialized.py tests/test_critic.py \
+              tests/test_report.py tests/test_anthropic_provider.py -v --no-cov
+```
+
+## Demoing each feature
+
+The demos below assume a working Ollama install with the model from
+`.wikifi/config.toml` (default `qwen3.6:27b`). If you want the hosted
+Anthropic path instead, set `ANTHROPIC_API_KEY` and pass
+`--provider anthropic` to the relevant commands; everything else is
+identical.
+
+### 1. Source-traceable citations + 5. Contradiction surfacing
+
+Run a walk against this repo:
+
+```bash
+make init                  # one-time; idempotent
+make walk
+```
+
+Open `.wikifi/<section>.md` for any populated primary section
+(`entities.md`, `capabilities.md`, `cross_cutting.md`, …). At the bottom
+you should see:
+
+```
+## Sources
+1. `wikifi/extractor.py:115-187`
+2. `wikifi/aggregator.py:54-79`
+…
+```
+
+Where the aggregator detected disagreement across files, the section
+also carries a `## Conflicts in source` block enumerating each
+position with its sources. Search for it via:
+
+```bash
+rg -n '^## Conflicts in source' .wikifi/
+```
+
+(For unit-level evidence: `tests/test_evidence.py` exercises citation
+rendering and contradiction rendering directly; `tests/test_aggregator.py`
+covers the end-to-end "claim → SourceRef" resolution.)
+
+### 2. Incremental walks (content-addressed cache) + 11. Resumability
+
+Run a walk, then run it again immediately:
+
+```bash
+make walk                  # first walk: extracts every in-scope file
+make walk                  # second walk: cache_hits == files_seen
+```
+
+The second invocation prints `cache_hits=N` in the **Extraction** row
+of the walk report — that's the number of files served from the cache
+without an LLM call.
+
+To force a clean re-walk:
+
+```bash
+uv run wikifi walk --no-cache
+```
+
+Resumability is the same mechanism: the cache is persisted after every
+file finishes, so a `Ctrl-C` mid-walk loses no progress — the next
+`wikifi walk` continues from the file that was in flight when the
+crash happened.
+
+(Unit evidence: `tests/test_cache.py`, plus
+`test_run_walk_persists_cache_for_resumability` in
+`tests/test_orchestrator.py`.)
+
+### 3. Cross-file context (import graph)
+
+Open the live extraction prompt for any application file. The walker
+includes a `Neighbor files` block listing files this one imports from
+or is imported by:
+
+```bash
+uv run wikifi walk -v 2>&1 | rg -A3 "Neighbor files" | head
+```
+
+You can also inspect the graph directly:
+
+```python
+from pathlib import Path
+from wikifi.repograph import build_graph
+from wikifi.walker import WalkConfig, iter_files
+
+config = WalkConfig(root=Path("."))
+files = list(iter_files(config))
+graph = build_graph(repo_root=Path("."), files=files)
+node = graph.get("wikifi/aggregator.py")
+print(node.imports)        # ('wikifi/cache.py', 'wikifi/evidence.py', ...)
+print(node.imported_by)    # ('wikifi/orchestrator.py', ...)
+```
+
+(Unit evidence: `tests/test_repograph.py`, plus
+`test_extract_repo_injects_neighbor_context_when_graph_supplied` in
+`tests/test_extractor.py`.)
+
+### 4. Type-aware extractors (SQL / OpenAPI / Protobuf / GraphQL / migrations)
+
+Drop a SQL file into a target project and run a walk:
+
+```bash
+mkdir -p /tmp/demo && cd /tmp/demo && git init -q
+cat > schema.sql <<'EOF'
+CREATE TABLE customer (
+  id INTEGER PRIMARY KEY,
+  email VARCHAR(255) UNIQUE NOT NULL
+);
+CREATE TABLE orders (
+  id INTEGER PRIMARY KEY,
+  customer_id INTEGER REFERENCES customer(id),
+  total INTEGER NOT NULL
+);
+CREATE INDEX idx_orders_customer ON orders (customer_id);
+EOF
+uv run --project /home/user/wikifi wikifi init
+uv run --project /home/user/wikifi wikifi walk
+```
+
+The walk report's **Extraction** row shows `specialized=1`. The
+findings produced for `entities.md`, `integrations.md` (the FK), and
+`cross_cutting.md` (the index + UNIQUE invariants) come from the
+deterministic SQL parser — no LLM call was made for `schema.sql`.
+
+The same routing covers `*.proto`, `*.graphql`, OpenAPI YAML / JSON
+specs, and any SQL file under `migrations/` / `alembic/` /
+`db/migrate/` directories.
+
+(Unit evidence: `tests/test_specialized.py` covers each parser;
+`test_extract_repo_routes_sql_through_specialized_extractor` in
+`tests/test_extractor.py` covers the end-to-end routing.)
+
+### 6. Critic + reviser pass on derivatives
+
+Re-run the walk with `--review`:
+
+```bash
+uv run wikifi walk --review
+```
+
+The walk report shows `sections_revised=N` in the **Derivation** row —
+that's how many derivative sections (personas / user stories /
+diagrams) the critic flagged as below the score threshold and the
+reviser improved.
+
+(Unit evidence: `tests/test_critic.py` covers the critic loop and the
+"only accept revision if it scores at least as well" guard. Integration:
+`test_run_walk_review_flag_invokes_critic`.)
+
+### 8. Coverage + quality report
+
+After a walk:
+
+```bash
+uv run wikifi report           # purely structural; no LLM calls
+uv run wikifi report --score   # adds critic-derived quality scores
+```
+
+Output is a markdown table of every section (files contributing,
+findings count, body size, score, headline gap):
+
+```
+| Section | Files | Findings | Body | Score | Headline gap |
+| --- | --- | --- | --- | --- | --- |
+| `entities` | 12 | 47 | 5132 | 9/10 | — |
+| `cross_cutting` | 4 | 9 | 1421 | 6/10 | unsupported: rate-limit policy |
+…
+```
+
+(Unit evidence: `tests/test_report.py`.)
+
+### 9. Anthropic provider with prompt caching
+
+Set the API key and switch the provider for a walk:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+WIKIFI_PROVIDER=anthropic uv run wikifi walk
+# or:
+uv run wikifi walk --provider anthropic
+```
+
+The provider sets `cache_control: {"type": "ephemeral"}` on the system
+prompt block. After the first per-file extraction call writes the
+cache, subsequent calls within the cache window read it for ~10% of
+the input price.
+
+To verify caching is active in the wild, intercept the SDK's response:
+
+```python
+from wikifi.providers.anthropic_provider import AnthropicProvider
+provider = AnthropicProvider(model="claude-opus-4-7", think="high")
+# After two calls with the same system prompt:
+# response.usage.cache_read_input_tokens > 0
+```
+
+(Unit evidence: `tests/test_anthropic_provider.py` locks in the
+`cache_control` placement, the `messages.parse` structured-output
+contract, the thinking → effort translation, and the APIError →
+RuntimeError mapping. `test_build_provider_returns_anthropic_when_selected`
+in `tests/test_orchestrator.py` covers dispatch.)
+
+## Tearing down
+
+The premium-pipeline state lives entirely under `.wikifi/`:
+
+```
+.wikifi/
+  config.toml
+  *.md                    # rendered sections (committable)
+  .notes/                 # per-section JSONL findings (gitignored)
+  .cache/                 # extraction + aggregation caches (gitignored)
+```
+
+Delete `.wikifi/.cache/` to drop the cache and force a full re-walk;
+delete the whole directory to start over.

pyproject.toml

@@ -14,6 +14,7 @@ dependencies = [
     "typer>=0.12",
     "rich>=13.7",
     "pathspec>=0.12",
+    "anthropic>=0.40",
 ]
 
 [project.scripts]

tests/test_aggregator.py

@@ -1,6 +1,12 @@
-from wikifi.aggregator import SectionBody, aggregate_all
+from wikifi.aggregator import (
+    AggregatedClaim,
+    AggregatedContradiction,
+    SectionBody,
+    aggregate_all,
+)
+from wikifi.cache import WalkCache, hash_section_notes
 from wikifi.sections import PRIMARY_SECTIONS
-from wikifi.wiki import WikiLayout, append_note, initialize
+from wikifi.wiki import WikiLayout, append_note, initialize, read_notes
 
 
 def _setup(tmp_path):
@@ -68,3 +74,88 @@ def raiser(schema, system, user):
     body = layout.section_path(section).read_text()
     assert "Aggregation failed" in body
     assert "Order line item." in body  # raw notes preserved
+
+
+def test_aggregate_renders_citations_and_contradictions(tmp_path, mock_provider_factory):
+    layout = _setup(tmp_path)
+    section = PRIMARY_SECTIONS[0]
+    append_note(
+        layout,
+        section,
+        {
+            "file": "a.py",
+            "summary": "domain",
+            "finding": "Tax computed at order time.",
+            "sources": [{"file": "src/order.py", "lines": [10, 25], "fingerprint": "abc"}],
+        },
+    )
+    append_note(
+        layout,
+        section,
+        {
+            "file": "b.py",
+            "summary": "domain",
+            "finding": "Tax computed at invoice time.",
+            "sources": [{"file": "src/invoice.py", "lines": [5, 12], "fingerprint": "def"}],
+        },
+    )
+
+    structured = SectionBody(
+        body="The system computes tax somewhere.",
+        claims=[AggregatedClaim(text="Tax computation lives at the boundary.", source_indices=[1, 2])],
+        contradictions=[
+            AggregatedContradiction(
+                summary="Where tax is computed.",
+                positions=[
+                    AggregatedClaim(text="At order time.", source_indices=[1]),
+                    AggregatedClaim(text="At invoice time.", source_indices=[2]),
+                ],
+            )
+        ],
+    )
+
+    provider = mock_provider_factory(
+        json_factory=lambda schema, system, user: structured,
+    )
+    aggregate_all(layout=layout, provider=provider)
+    body = layout.section_path(section).read_text()
+    assert "Conflicts in source" in body
+    assert "src/order.py:10-25" in body
+    assert "src/invoice.py:5-12" in body
+    assert "## Sources" in body
+
+
+def test_aggregate_uses_cache_to_skip_unchanged_notes(tmp_path, mock_provider_factory):
+    layout = _setup(tmp_path)
+    section = PRIMARY_SECTIONS[0]
+    append_note(layout, section, {"file": "a.py", "summary": "x", "finding": "Order entity."})
+
+    cache = WalkCache()
+    notes_hash = hash_section_notes(read_notes(layout, section))
+    cache.record_aggregation(
+        section.id,
+        notes_hash=notes_hash,
+        body="Cached body for the section.",
+        claims=[],
+        contradictions=[],
+    )
+
+    provider = mock_provider_factory()  # no responses queued — must not be called
+    stats = aggregate_all(layout=layout, provider=provider, cache=cache)
+
+    body = layout.section_path(section).read_text()
+    assert "Cached body for the section." in body
+    assert stats.sections_cached == 1
+
+
+def test_aggregate_records_cache_entry_after_synthesis(tmp_path, mock_provider_factory):
+    layout = _setup(tmp_path)
+    section = PRIMARY_SECTIONS[0]
+    append_note(layout, section, {"file": "a.py", "summary": "x", "finding": "Order."})
+
+    cache = WalkCache()
+    provider = mock_provider_factory(
+        json_factory=lambda schema, system, user: SectionBody(body="Synthesized body."),
+    )
+    aggregate_all(layout=layout, provider=provider, cache=cache)
+    assert section.id in cache.aggregation