fix(renderer): stop shipping an unclearable wall as a success

[us]

Follow-up to #494. That PR fixed a real bug but not this one, and prod proved it: after #494 reached the box, the same request still came back as a billed success.

The case, live on prod today

POST /v1/scrape {"url":"https://www.prlib.ru/en/history/619410"}

success: true            creditCost: 1
markdown: "# Security Check\n\nChecking your browser before accessing the site\n\nThis will take a few seconds"   (95 chars, entire body)
metadata.renderedWith: "lightpanda"
renderDecision: {"kind":"failover","chain":["lightpanda","chrome","chrome_proxy"],"reason":"other"}

A customer paid a credit for an interstitial, and an agent consuming this reads "Checking your browser" as the page.

Why #494 did not catch it

#494 threaded the renderer's truncation flag into the wall detector, because a budget-truncated body has no closing </body> and was extracting as an empty string. Correct, and it stands. But it is not what was happening here: I pulled the real rawHtml (5,462 chars) and it has a closing </body>, 77 visible characters, and contains two phrases from the detector's list. So looks_like_generic_bot_wall returns true for this input with or without that flag.

Which meant the wall was being detected, and something downstream ignored the answer.

The actual cause

FallbackRenderer::fetch_with_js, the tail after the ladder is exhausted:

// Return the best thin result if we have one, otherwise the last error.
if let Some(mut result) = thin_result {
    result.credit_cost = credit_for(last);
    result.render_decision = Some(RenderDecision::Failover { chain, reason });
    ...
    Ok(result)

Every tier classified the body and rejected it — that rejection is exactly why it sits in thin_result rather than having been accepted. The tail then discards the verdict, stamps a page credit and returns it as a success. The existing comment above the pinned-renderer hint already names the symptom: "silently returning broken markdown with success: true". That hint only covers the hard-pinned single-tier case; the auto-failover path had nothing.

The tail now re-reads the verdict and fails the request.

Scope, deliberately narrow

Only the generic phrase-list wall converts to an error. A Cloudflare challenge or a named vendor shell keeps shipping as a thin result, because those have their own recovery arms (cloak, chrome_proxy) that read the body back — cloak_recover_on_cf_false_skips_arm_on_exhausted_deadline pins that shape and my first, broader attempt broke it. A thin but wall-free page still ships, which is what the recall invariant rests on.

Verification

• cargo test --workspace: 4226 passed
• cargo test -p crw-renderer --features camoufox,cloak,cdp: 1024 passed (the ladder tail lives behind those default-off features for two of the tiers)
• clippy clean in both configurations, fmt clean
• Two new tests, one per half of the gate. Disabling the guard fails the wall test and leaves the thin test passing, so each pins what it claims to.

Merging reaches prod within the hour; I will re-run the live case against the new engine and report the result on this PR.

[us]

Why the per-attempt gate never ran

An independent trace of this request confirms the tail is the only path, and explains the piece I had left open — why the accept gate with && !is_bot_wall did not fire:

1. The chrome hedge fires: lightpanda and chrome race concurrently. classify_js_attempt evaluates both, and its acceptable flag does check looks_like_generic_bot_wall. Neither tier was acceptable.
2. try_hedge therefore returns HedgeOutcome::Thin, carrying the richer of the two bodies (max_by_key(html.len())) — lightpanda's 5,462-char wall shell. That becomes thin_result, and hedge_done = true.
3. The serial for renderer in renderers loop opens with if hedge_done { break; }. The accept gate lives inside that loop, so on a hedged request it is never reached. It gates individual attempts, not the surviving result.
4. saw_hard_block was set (the wall), so the chrome_proxy recovery arm fired and pushed ChromeProxy onto the chain before its fetch — which is why the chain shows three tiers. Its own result was re-validated and was not better, so thin_result stayed lightpanda's, which is why renderedWith is still "lightpanda".
5. last_failover_reason is only assigned in the serial loop's error branch, which never ran, so it defaulted to Other — matching the observed "reason":"other".

Every tier that touched this body had already flagged it as a wall. The gap is that nothing re-read that verdict before the final Ok(result), so it shipped as "the best of what we tried". That is exactly where this patch sits.

It also independently confirms the truncation flag is orthogonal here: both <body> tags were present, so #494's parameter could not have suppressed or caused this match either way.

[us]

Confirmed on prod once #497 landed alongside it: the wall now returns success:false with no credit and no markdown. This PR's guard was firing all along (its message appears verbatim in the engine log), but the HTTP-shell fallback one layer up was substituting the same wall back in. #497 closes that. Verification detail is on #497.