fix(renderer): keep the HTTP shell from resurrecting a refused wall

[us]

Third and final link in the chain #494 → #496 → this. Each fix was correct and each time prod showed the wall still coming back, because the next layer down handed it over again.

The evidence

#496 made the ladder tail refuse a wall. It works — the engine's own log on prod says so:

ERROR crw_renderer: JS escalation failed for soft-block status; surfacing HTTP shell
with warning: blocked by an anti-bot wall that none of the 3 renderer tier(s)
attempted could clear      status_code=403

Read that line carefully: my guard fired, returned the error, and the caller then surfaced the HTTP shell anyway. The shell is the same wall, so the customer still got success: true and a credit.

The fix

When the JS ladder fails, the HTTP body substitutes for it so the caller gets content instead of nothing. Both fallback arms already decline to substitute in two cases: the caller pinned a renderer, or asked for a screenshot — contracts a silent substitution would break.

A wall belongs in the same set, for the same reason: it is not a fallback, it is the failure repeated. Added to the existing guard expression at both sites rather than as a new branch.

A shell that is not a wall still substitutes exactly as before, so the recall this fallback exists to protect is untouched.

Verification

• cargo test --workspace: 4227 passed
• cargo test -p crw-renderer --features camoufox,cloak,cdp: 1025 passed
• clippy clean in both configurations, fmt clean
• New test http_shell_fallback_does_not_resurrect_a_wall; disabling both guards makes it fail, so it pins the behaviour rather than passing incidentally

Why this took three PRs

Worth recording, since the chain is instructive:

• fix(renderer): report scrape outcomes truthfully #494 threaded the renderer truncation flag into the detector. A real bug — a budget-truncated body has no closing </body> and extracted as an empty string — but not this symptom. I diagnosed it from the truncated: true flag in the live response without checking whether the body actually lacked the tag. It did not.
• fix(renderer): stop shipping an unclearable wall as a success #496 found the ladder tail returning a rejected body as "the best of what we tried". Also real, also not sufficient on its own.
• This is the layer that was undoing both.

I verified each one against prod after it deployed rather than assuming, which is the only reason the second and third links were found at all.

[us]

Verified on prod

Engine pin c74bcf365a72, container restarted, health warmed. Same URL that started this:

POST /v1/scrape {"url":"https://www.prlib.ru/en/history/619410"}

              before          after
success       true            false
creditCost    1               none      <- the customer is no longer billed
markdown      95 chars of     none      <- the interstitial no longer ships as content
              "Security Check
               Checking your
               browser..."

Regression check, same run

URL	success	markdown	credit	tier
example.com	true	167 ch	1	http
en.wikipedia.org/wiki/Web_scraping	true	31,112 ch	1	http
github.com/us/crw	true	8,629 ch	1	http
news.ycombinator.com	true	11,035 ch	1	http

example.com at 167 characters is the live counterpart of the thin_but_real_page_still_ships_from_the_ladder_tail unit test: a short but genuine page is unaffected, which is what the recall invariant rests on.

One loose end

The error the customer sees is generic: "The fetch did not complete. This can be the target or our egress." The engine says blocked by an anti-bot wall that none of the N renderer tier(s) attempted could clear, but the SaaS error mapping flattens it. Worth surfacing the real reason so a caller can tell "the site blocked us" from "our egress failed" — separate change, not in this PR.