fix(v2): carry llmUsage through to_v2_document so the SaaS can settle its reserve

[behramcelen]

The bug

V2Document (crates/crw-server/src/routes/v2/adapters.rs) had no llm_usage field, and to_v2_document() never copied one. Every /v2/scrape response dropped the token telemetry that /v1 has always carried on ScrapeData::llm_usage.

That field is billing infrastructure, not diagnostics. The SaaS prices a managed-LLM scrape by reserving a worst-case estimate up front and settling it against the engine's reported usage. With the field absent there is nothing to settle against, so it holds the reserve — and the caller pays the worst case on every request, forever.

Measured on production

Same page, same schema, same work:

Route	llmUsage	Charged
/v1/scrape + jsonSchema	present (631 in / 45 out)	5 credits
/v2/scrape object-form [{type:"json",schema}]	absent	91 credits

An 18x over-charge decided by nothing but which API version the caller used. Reproduced end to end through the SaaS with a temporary key; the resulting UsageEvent row for the v2 call is llmInputTokens: 0, llmModel: "crw-managed-pro", creditsCharged: 91 — the SaaS defaults, i.e. the "no telemetry" fallback branch.

Why the blast radius is the worst possible

The SaaS next.config.ts serves this v2 surface at the domain root specifically so FirecrawlApp(api_url="https://api.fastcrw.com") is a drop-in target for the firecrawl-py SDK.

So every customer migrating from Firecrawl — the exact audience that compatibility was built to win — landed on /v2 by default and was billed ~18x for it. One of them derived the ratio from his own invoice and reported it as "a 18x cost bump" before we found it internally.

The change

Add llm_usage to V2Document and copy it in to_v2_document().

Additive to the frozen Firecrawl envelope, which this file already does deliberately elsewhere for exactly this reason — V2CrawlStatus.blocked carries the comment "Additive to the Firecrawl envelope (their SDKs ignore unknown keys) and load-bearing: the SaaS bills off completed".

skip_serializing_if = "Option::is_none" keeps the key out of every non-LLM response, so callers who never trigger a model see the Firecrawl shape byte-identical to today.

Tests

Two regressions in routes::v2::adapters::tests:

1. v2_document_carries_llm_usage_for_the_saas_to_bill_on — asserts the usage survives the mapping and reaches the wire as llmUsage.inputTokens, which is the exact key the SaaS reads. A field that exists on the struct but serialises under another name would still be a silent 18x.
2. v2_document_omits_llm_usage_when_no_llm_ran — asserts a non-LLM scrape does not grow the key (no "llmUsage": null).

cargo test -p crw-server --lib routes::v2::adapters
test result: ok. 39 passed; 0 failed

Whole crw-server suite unchanged: parse_pdf_upload_returns_markdown fails identically with and without this commit (verified by stashing), and is unrelated — it is a PDF trailer parse failure.

Follow-up, not in this PR

Affected customers need refunding for the period this was live. The SaaS side should also grow a safety net so a missing-telemetry response can never again silently bill the worst case in full.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RUNZ9Nm19QSLWf4GhWpayq

[github-actions]

Thanks for the pull request. Before we can merge it, please read our Contributor License Agreement and sign it by posting the comment below in this PR.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}

[us]

Superseded by 538 (same branch, same commit).