Phase 0: evidence primitives, conformance CI gate, FRAMES bench harness - #199
Merged
Conversation
Add a conformance job that builds crw, starts the server, and diffs each response's shape against the committed Firecrawl v2 golden fixtures. Catches accidental removal/rename of a contract field; additive fields stay invisible. The live-search case is excluded via CONFORMANCE_SKIP because it depends on a third-party SearXNG fanout that rate-limits CI IPs to zero results and would flake the gate on an external dependency rather than on crw.
Add crw-core::evidence with Highlight, EvidenceCitation, SourceEvidence, Basis, SourcePolicy and ConfidenceLevel. These give answers, ranked results, and extracted fields a traceable basis (source, exact char span, content hash) for re-verification against the canonical scraped markdown. All types are additive and serde-stable (camelCase, skip_serializing_if), so attaching evidence to an existing response is byte-invisible to clients that ignore it. Wiring lands incrementally in the search and extract layers.
… judge Replace three hand-rolled UNTRUSTED fences (answer, summary, judge) with a single audited crw-extract::untrusted::wrap. The change judge previously used a fixed UNTRUSTED_DIFF delimiter with no nonce, so scraped diff content could forge the closing marker and break out of the fence; it now uses the same nonce-bearing delimiter as the others. Each call site's system prompt is updated to describe the new delimiter atomically.
Add ScrapeData.source_hash, the hex SHA-256 of the normalized canonical markdown (crw_diff::snapshot::hash_markdown), set at the single scrape choke point (crw-crawl::single::scrape_url) so single/crawl/batch all get it. Lets clients dedup/cache and ties evidence offsets to an exact source revision. Computed in crw-crawl (where crw-diff is a dep) rather than crw-extract, keeping the diff engine free of the extractor. The v2 adapter intentionally does not carry it, so the Firecrawl-compat surface is unchanged.
Add `crw bench`: runs a QA dataset (FRAMES, auto-downloaded from google/frames-benchmark or supplied via --dataset-file) through a SearchProvider (crw's /v1/search answer path), grades each answer with an LLM judge at temperature 0, and writes a reproducible snapshot (results jsonl + report json/md) with a seeded bootstrap 95% CI on the pass rate. A local/release tool, not a CI gate: it needs a running server (SearXNG + an LLM), a judge key, and network access to fetch the dataset.
The `url` positional and `--reset` flag declared `conflicts_with = "command"`, but a `#[command(subcommand)]` field is not a conflictable arg id, so clap's debug-build assertion aborted every `crw` invocation built without optimizations (release was unaffected — clap skips the assert there). Express the intent at the container level instead with `args_conflicts_with_subcommands = true`, so default-scrape-mode args are mutually exclusive with subcommands and debug builds run again.
- send scrapeOptions + answerTemperature so the server actually runs answer synthesis; without it the search route skips synthesis and returns an empty answer, so the harness scored ~0 regardless of quality - add --multi-round / --query-expand to A/B the floor vs tuned search config; both are recorded in report.json so runs are self-describing - add --concurrency (buffer_unordered) to overlap the I/O-bound questions; ceiling is upstream (SearXNG/proxy/TPM), so the safe value is empirical - persist each verdict to frames_results.jsonl as it lands instead of buffering to the end, so a multi-hour run survives a crash/kill Claude-Session: https://claude.ai/code/session_01HncCgCTJVcVLpLLgJ5gPki
us
force-pushed
the
feat/phase0-conformance-ci
branch
from
July 1, 2026 19:34
2efb34c to
d81d477
Compare
There was a problem hiding this comment.
Pull request overview
Adds Phase-0 foundations for provenance/evidence and contract stability, plus new tooling (benchmark harness) and a CI conformance gate to prevent silent Firecrawl v2 API shape regressions. Also hardens prompt-injection defenses by consolidating untrusted-content fencing behind a single nonce-based primitive.
Changes:
- Introduces shared evidence/provenance primitives in
crw-coreand addssourceHashto scrape responses (computed at the crawl scrape “choke point”). - Replaces ad-hoc untrusted fencing in extract/summary/judge/answer with
crw-extract::untrusted::{random_nonce, wrap}(nonce-bearing delimiters). - Adds FRAMES benchmark harness (
crw bench) and a CI conformance job that diffs response shape against committed golden fixtures (with selective case skipping).
Reviewed changes
Copilot reviewed 19 out of 20 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/crw-server/src/routes/v2/adapters.rs | Updates v2 adapter tests to include new source_hash field in ScrapeData. |
| crates/crw-extract/tests/judge_tests.rs | Updates judge request-fencing test to match the new nonce-bearing delimiter format. |
| crates/crw-extract/src/untrusted.rs | Adds the single audited primitive for nonce-fenced untrusted-content wrapping. |
| crates/crw-extract/src/summary.rs | Switches summary prompt construction to use untrusted::wrap + shared nonce generation. |
| crates/crw-extract/src/lib.rs | Ensures extractor-produced ScrapeData includes the new source_hash field (left unset here). |
| crates/crw-extract/src/judge.rs | Uses untrusted::wrap for change-judge diffs and updates prompt/test wording accordingly. |
| crates/crw-extract/src/answer.rs | Uses untrusted::wrap for per-source blocks in answer synthesis; removes local nonce helper. |
| crates/crw-crawl/src/single.rs | Computes source_hash from canonical markdown at the scrape choke point. |
| crates/crw-crawl/src/pdf.rs | Initializes source_hash to None for PDF scrape data (hash filled later when applicable). |
| crates/crw-core/tests/types_tests.rs | Extends serialization tests to account for source_hash. |
| crates/crw-core/src/types.rs | Adds source_hash to ScrapeData (camelCase on wire via struct-level serde). |
| crates/crw-core/src/lib.rs | Exposes the new evidence module from crw-core. |
| crates/crw-core/src/evidence.rs | Adds evidence/provenance primitives (Highlight, EvidenceCitation, Basis, SourcePolicy, etc.). |
| crates/crw-cli/src/main.rs | Fixes clap arg/subcommand conflict wiring and adds the bench subcommand. |
| crates/crw-cli/src/commands/mod.rs | Registers the new bench command module. |
| crates/crw-cli/src/commands/bench.rs | Adds FRAMES benchmark harness (dataset fetch, concurrent runs, LLM judge, snapshots + bootstrap CI). |
| conformance/conformance/compare.py | Adds env-driven case skipping for conformance runs (used to avoid flaky live-search in CI). |
| conformance/.gitignore | Ignores uv lockfile (uv.lock) under conformance. |
| .gitignore | Ignores bench datasets and run artifacts. |
| .github/workflows/ci.yml | Adds Firecrawl v2 conformance gate job that runs compare against a local crw serve. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+542
to
+546
| let mut jsonl = String::new(); | ||
| for r in results { | ||
| jsonl.push_str(&serde_json::to_string(r).map_err(|e| e.to_string())?); | ||
| jsonl.push('\n'); | ||
| } |
Comment on lines
1
to
3
| .venv/ | ||
| uv.lock | ||
| __pycache__/ |
Comment on lines
+102
to
+105
| assert!( | ||
| body.contains("=====UNTRUSTED:DIFF:") && body.contains("=====/UNTRUSTED:DIFF:"), | ||
| "diff must be fenced" | ||
| ); |
Comment on lines
+154
to
+155
| assert!(p.contains("=====UNTRUSTED:DIFF:")); | ||
| assert!(p.contains("=====/UNTRUSTED:DIFF:")); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of the data-layer expansion: the provenance + security + measurement
foundation, plus a CI guard so the work that builds on it can't silently break
the API contract. All changes are additive and the Firecrawl-compat
/v2surface is byte-unchanged.
What's in here (6 commits)
shape against committed golden fixtures; catches accidental removal/rename of
a contract field. Live-search excluded (third-party SearXNG fanout flakes on
CI IPs); the rest hit stable targets and are deterministic.
Highlight,EvidenceCitation,SourceEvidence,Basis,SourcePolicy,ConfidenceLevel) — the shared vocabulary for traceable answers/fields.untrusted::wrap; the change judge previously had a fixed, forgeabledelimiter and is now nonce-fenced like answer/summary.
sourceHashon/v1scrape responses (hex SHA-256 ofthe normalized canonical markdown), so clients can dedup/cache and evidence
offsets tie to an exact source revision.
crw bench— reproducible FRAMES QA harness (auto-download,LLM judge at temp 0, seeded bootstrap 95% CI, run snapshots).
crw(release was unaffected).Validation
cargo test --workspace: 1106 passed, 0 failedcargo clippy --workspace --all-targets -D warnings: clean ·cargo fmt --check: cleancompare: exit 0 (Tier-1 shape-match, search excluded)sourceHashverified e2e on/v1: 64-hex, equalshash_markdown(markdown)crw benchsmoke-tested end-to-end (plumbing + snapshot writing)Not in scope / deferred
run needs a live server (SearXNG + LLM) + network and is the next step.
(Phase 2 re-verification/dedup) — would be write-only today.
/v1/search) is Phase 1a.Builds on
origin/main;Cargo.lock's pre-existing internal-version stalenessis intentionally left untouched.