Add preprocess-legal plugin (slice 1: text-native extraction + confidence gate)

external_plugins/preprocess-legal/.claude-plugin/config-template.md

@@ -0,0 +1,35 @@
+# preprocess-legal — configuration
+
+The engine reads a JSON config. Point the hook at it with the
+`PREPROCESS_LEGAL_CONFIG` environment variable, or place it at the default path:
+
+```
+~/.claude/plugins/config/claude-for-legal/preprocess/config.json
+```
+
+## Example
+
+```json
+{
+  "enabled": true,
+  "threshold": 0.65,
+  "min_chars_per_page": 100,
+  "file_types": [".pdf"],
+  "min_bytes_to_route": 0,
+  "cache_dir": ""
+}
+```
+
+## Fields
+
+| Field | Default | Meaning |
+|-------|---------|---------|
+| `enabled` | `false` | Master switch. Opt-in — while `false`, every Read passes through untouched. |
+| `threshold` | `0.65` | Confidence at or above this passes through as clean text; below it, the document is withheld. |
+| `min_chars_per_page` | `100` | Characters a page must clear to count as "covered" text (separates a real page from a scanned image's stray characters). |
+| `file_types` | `[".pdf"]` | Only these suffixes are intercepted. Everything else is passed through. |
+| `min_bytes_to_route` | `0` | Files smaller than this are read directly (not worth extraction latency). `0` disables the floor. |
+| `cache_dir` | `""` | Where extracted text / withheld markers are written. Empty → a `.preprocess-cache/` folder beside this config. |
+
+A missing or malformed config falls back to the safe default (disabled) — a
+broken config never breaks your Read tool.

external_plugins/preprocess-legal/.claude-plugin/plugin.json

@@ -0,0 +1,9 @@
+{
+  "name": "preprocess-legal",
+  "version": "0.1.0",
+  "description": "Transparent local pre-processing for document-heavy legal work. A PreToolUse hook intercepts each Read of a PDF, extracts text-native content locally, scores per-document extraction confidence, and redirects the read to clean text on a pass or to an honest withheld-marker on a fail. Refuses to pass degraded, plausible-looking extractions into context silently. Opt-in. Slice 1: text-native extraction + confidence gate (OCR fallback deferred).",
+  "author": {
+    "name": "Eric Tetzlaff",
+    "email": "eric@erictetzlaff.com"
+  }
+}

external_plugins/preprocess-legal/.gitignore

@@ -0,0 +1,11 @@
+# Materialized extraction cache (clean text + withheld markers). Local per-user,
+# never shipped.
+.preprocess-cache/
+
+# Python
+__pycache__/
+*.py[cod]
+.pytest_cache/
+
+# Local machine settings that must never ship with the plugin
+.claude/settings.local.json

external_plugins/preprocess-legal/README.md

@@ -0,0 +1,74 @@
+# preprocess-legal
+
+Transparent local pre-processing for document-heavy legal work. Addresses
+[issue #100](https://github.com/anthropics/claude-for-legal/issues/100): when
+Claude reads a folder of PDFs (court filings, contracts, scanned exhibits), each
+file is read in its native form — slow and token-heavy for large, scanned, or
+poorly-OCR'd matters.
+
+This plugin intercepts each read, extracts the text locally, and hands the model
+clean text instead of the raw PDF — **but only when the extraction is
+trustworthy.** A degraded scan is never silently passed through as hollow text.
+
+## How it works
+
+A `PreToolUse` hook fires on every `Read`. For a routed PDF it:
+
+1. **Routes** — is this file in scope? (opt-in, by type and optional size floor)
+2. **Extracts** — text-native content via `pdfplumber`
+3. **Scores** — per-document confidence, as an explainable breakdown:
+   - `coverage` — fraction of pages that produced real text
+   - `density` — mean characters per page vs. a text-native baseline
+   - `legibility` — fraction of characters that are actually readable
+   - `overall = (weighted coverage + density) × legibility` — legibility is a
+     **quality veto**, so a page full of plausible-looking mojibake cannot pass
+     on quantity alone
+4. **Gates** — at/above `threshold` → pass; below → withhold
+5. **Redirects** — rewrites the Read's `file_path` (`updatedInput`) to point at:
+   - a **clean-text** file on a pass, or
+   - an **honest withheld-marker** on a fail — text that *says* the document was
+     held back and why, never hollow content masquerading as the document
+
+Because the engine writes the exact bytes the model reads, "refuse to pass
+degraded output silently" is literal, not advisory. The redirect is transparent
+to the model (confirmed on Claude Code ≥ v2.1.190, CLI + Cowork + Desktop).
+
+## Design notes
+
+- **Per-file, not per-folder.** Hooks fire per tool call — reading 20 PDFs fires
+  20 times, each with one path. Routing is per document.
+- **Fails safe.** A missing/broken config, a missing `pdfplumber`, or an
+  encrypted/corrupt PDF never breaks a Read — the hook either withholds (fail
+  closed) or passes the original through (fail open on unexpected errors).
+- **Swappable backend.** Only `engine/extract.py` imports `pdfplumber`. The
+  scoring and gate logic operate on a plain `ExtractionResult`, so slice 2's OCR
+  cascade plugs in behind the same interface without touching the gate.
+- **Warm cache.** Extraction is synchronous (the redirect target must exist when
+  the read returns). Re-reading an unchanged file hits a cache keyed on
+  path + size + mtime.
+
+## Status
+
+**Slice 1** — text-native extraction + confidence gate + withheld-marker.
+**Deferred:** OCR fallback cascade for low-confidence scans (slice 2), caching
+tuning for heavy synchronous matters (slice 3).
+
+## Install & enable
+
+1. Install the extraction dependency: `pip install -r requirements.txt`
+2. Create a config (see `.claude-plugin/config-template.md`) and set `enabled: true`
+3. Point the hook at it via `PREPROCESS_LEGAL_CONFIG`, or use the default path
+
+## Develop
+
+```
+python -m pytest tests/     # pure scoring + gate tests, no pdfplumber required
+```
+
+## Open questions (tracked on issue #100)
+
+- Scoped to a practice-area plugin, or closer to Cowork itself? Built behind a
+  clean interface either way.
+- Real-matter tuning: scanned-vs-text ratio, latency tolerance, and which
+  structural losses (tables, exhibit stamps, Bates numbers, redactions) matter
+  most — awaiting the reporter's answers.

external_plugins/preprocess-legal/engine/__init__.py

@@ -0,0 +1,26 @@
+"""preprocess-legal extraction engine.
+
+Slice 1: text-native PDF extraction + per-document confidence scoring + a gate
+that refuses to silently pass degraded output into the model's context.
+
+The engine is deliberately split so the parts that need the pdfplumber dependency
+(``extract``) are isolated from the pure decision logic (``confidence``, ``gate``).
+Only ``extract`` imports pdfplumber; everything else operates on plain data and is
+unit-testable without any PDF library installed. This keeps the extraction backend
+swappable (the interface, not pdfplumber, is the contract) per the plugin's design.
+"""
+
+from .confidence import ConfidenceScore, score_extraction
+from .gate import Decision, ProcessResult, decide, process
+from .config import PreprocessConfig, load_config
+
+__all__ = [
+    "ConfidenceScore",
+    "score_extraction",
+    "Decision",
+    "ProcessResult",
+    "decide",
+    "process",
+    "PreprocessConfig",
+    "load_config",
+]

external_plugins/preprocess-legal/engine/confidence.py

@@ -0,0 +1,128 @@
+"""Per-document extraction confidence scoring.
+
+Pure functions over an ``ExtractionResult`` — no I/O, no PDF library. Given the
+text a backend pulled out of a document, decide how much to trust it.
+
+The score is deliberately a breakdown, not a single opaque number. A confidence
+value with no per-dimension reasons is not auditable, and the whole point of the
+gate is that an attorney (or a downstream task) can see *why* a document was held
+back. Each dimension is a 0..1 signal; ``overall`` is their weighted mean.
+
+Dimensions
+----------
+coverage    Fraction of pages that produced real text. A scanned exhibit run
+            through a text extractor returns empty pages — low coverage is the
+            loudest signal that this document is image-based, not text-native.
+density     Mean characters per page, saturated against a text-native baseline.
+            A page with 12 stray characters scores near zero; a full page of
+            body text saturates to 1.0.
+legibility  Fraction of extracted characters that are actually readable (letters,
+            digits, ordinary punctuation, whitespace) rather than control bytes
+            or Unicode replacement chars. Catches the dangerous case: text that
+            extracted to *something* but is mojibake — plausible-looking garbage
+            that nothing downstream would flag.
+"""
+
+from __future__ import annotations
+
+import string
+from dataclasses import dataclass
+from typing import Dict
+
+from .models import ExtractionResult
+
+# Characters we consider legible. Everything printable, plus whitespace.
+_LEGIBLE = set(string.printable)
+
+# A full page of body text runs well over a thousand characters. We saturate the
+# density signal here so that anything at or above a normal text page scores 1.0
+# and only sparse / near-empty pages are penalized.
+_DENSITY_SATURATION_CHARS = 1200.0
+
+# Coverage and density measure *quantity* — is there enough text on the page.
+# They are combined into a quantity score, coverage weighted higher because "the
+# extractor returned nothing for this page" is the highest-signal indicator of an
+# image-only document.
+_QUANTITY_WEIGHTS: Dict[str, float] = {
+    "coverage": 0.6,
+    "density": 0.4,
+}
+# Legibility measures *quality* — is the extracted text actually readable rather
+# than mojibake. It multiplies the quantity score rather than adding to it, so it
+# acts as a veto: a page full of illegible bytes has high quantity but cannot pass
+# on quantity alone. This is the defense against text that "extracted to something"
+# but is structurally garbage — the expensive way to be wrong in a case file.
+
+
+@dataclass
+class ConfidenceScore:
+    """A 0..1 overall score plus the per-dimension breakdown behind it."""
+
+    overall: float
+    dimensions: Dict[str, float]
+    page_count: int
+    mean_chars_per_page: float
+
+    def reason(self) -> str:
+        """One-line human explanation, used in the withheld marker."""
+        parts = [f"{k}={v:.2f}" for k, v in self.dimensions.items()]
+        return (
+            f"confidence={self.overall:.2f} "
+            f"({', '.join(parts)}; {self.page_count} pages, "
+            f"{self.mean_chars_per_page:.0f} chars/page)"
+        )
+
+
+def _legibility(text: str) -> float:
+    # Whitespace-only text carries no legible content — its "printable" newlines
+    # must not manufacture confidence for an effectively empty page.
+    if not text.strip():
+        return 0.0
+    legible = sum(1 for ch in text if ch in _LEGIBLE)
+    return legible / len(text)
+
+
+def score_extraction(
+    result: ExtractionResult,
+    *,
+    min_chars_per_page: int = 100,
+) -> ConfidenceScore:
+    """Score how trustworthy an extraction is.
+
+    ``min_chars_per_page`` is the bar a page must clear to count as "covered" —
+    below it, a page is treated as effectively blank (a scanned image yields a
+    handful of stray characters at most).
+    """
+    pages = result.page_texts
+    page_count = len(pages)
+
+    # A hard extraction failure or a document with no pages has no trustworthy
+    # text by definition. Return an all-zero score so the gate withholds it.
+    if not result.ok or page_count == 0:
+        dims = {"coverage": 0.0, "density": 0.0, "legibility": 0.0}
+        return ConfidenceScore(0.0, dims, page_count, 0.0)
+
+    covered = sum(1 for p in pages if len(p.strip()) >= min_chars_per_page)
+    coverage = covered / page_count
+
+    total_chars = sum(len(p) for p in pages)
+    mean_chars = total_chars / page_count
+    density = min(mean_chars / _DENSITY_SATURATION_CHARS, 1.0)
+
+    legibility = _legibility(result.full_text)
+
+    dims = {
+        "coverage": round(coverage, 4),
+        "density": round(density, 4),
+        "legibility": round(legibility, 4),
+    }
+    # Quantity (weighted coverage + density) vetoed by quality (legibility).
+    quantity = sum(_QUANTITY_WEIGHTS[k] * dims[k] for k in _QUANTITY_WEIGHTS)
+    overall = quantity * legibility
+
+    return ConfidenceScore(
+        overall=round(overall, 4),
+        dimensions=dims,
+        page_count=page_count,
+        mean_chars_per_page=round(mean_chars, 1),
+    )

external_plugins/preprocess-legal/engine/config.py

@@ -0,0 +1,94 @@
+"""Configuration for the preprocess-legal engine.
+
+Loaded from the practice profile (a JSON file the cold-start / customize flow
+writes). Everything has a safe default so the engine runs headless in tests and
+so a missing/partial config never crashes a Read — it just falls back to
+sensible behavior.
+
+The capability is opt-in: ``enabled`` defaults to False. Nothing is intercepted
+until the attorney turns it on.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import List, Optional
+
+
+@dataclass
+class PreprocessConfig:
+    # Master switch. Opt-in by design — an off config passes every Read through
+    # untouched.
+    enabled: bool = False
+
+    # Confidence at or above this passes through as clean text; below it, the
+    # document is withheld (slice 2 will route it to OCR instead).
+    threshold: float = 0.65
+
+    # A page must clear this many characters to count as "covered" text.
+    min_chars_per_page: int = 100
+
+    # Only these suffixes are routed. Anything else is passed through untouched.
+    file_types: List[str] = field(default_factory=lambda: [".pdf"])
+
+    # Files smaller than this are cheap enough to read directly — no point
+    # paying extraction latency to save a few tokens. 0 disables the floor.
+    min_bytes_to_route: int = 0
+
+    # Where extracted text / withheld markers are written. Resolved to an
+    # absolute path at load time. Defaults to a cache dir beside the config.
+    cache_dir: str = ""
+
+    def routes(self, path: Path) -> bool:
+        """Should this path be intercepted at all?"""
+        if not self.enabled:
+            return False
+        if path.suffix.lower() not in [s.lower() for s in self.file_types]:
+            return False
+        try:
+            if self.min_bytes_to_route and path.stat().st_size < self.min_bytes_to_route:
+                return False
+        except OSError:
+            return False
+        return True
+
+
+def load_config(config_path: Optional[str] = None) -> PreprocessConfig:
+    """Load config from JSON, falling back to defaults for any missing key.
+
+    A malformed or missing file yields the default (disabled) config rather than
+    raising — a broken config must never break the user's Read tool.
+    """
+    cfg = PreprocessConfig()
+
+    if config_path:
+        p = Path(config_path)
+        if p.is_file():
+            try:
+                data = json.loads(p.read_text(encoding="utf-8"))
+            except (ValueError, OSError):
+                data = {}
+            for key in (
+                "enabled",
+                "threshold",
+                "min_chars_per_page",
+                "file_types",
+                "min_bytes_to_route",
+                "cache_dir",
+            ):
+                if key in data:
+                    setattr(cfg, key, data[key])
+
+    # Resolve the cache dir. Default: a hidden cache folder next to the config,
+    # or under the current dir if no config path was given.
+    if cfg.cache_dir:
+        cache = Path(cfg.cache_dir).expanduser()
+    elif config_path:
+        cache = Path(config_path).expanduser().resolve().parent / ".preprocess-cache"
+    else:
+        cache = Path(".preprocess-cache")
+    cfg.cache_dir = str(cache)
+
+    return cfg