Heavily-protected sites (CARFAX / Cloudflare-class) blocked by client-side bot detection

[us]

Summary

Reported by @itsSteven_Phan: scraping a CARFAX vehicle-history report returns
"Looks like your internet browser isn't supported" instead of the content.

Repro: crw scrape "https://vhr.carfax.ca/?id=..."

What we ruled out (it is NOT the User-Agent)

The v0.18.0→v0.18.3 work fixed UA presentation across every path:

• #176 — HTTP-only fetcher → Chrome 150
• v0.18.2 — CDP path (Network.setUserAgentOverride)
• v0.18.3 — lightpanda (strips the Mozilla/5.0 prefix it rejects)

But CARFAX still blocks us. Evidence it's not UA-based:

• HTTP test: every UA (full Chrome, stripped, lightpanda, even curl) gets
  the same 200 + identical 22 KB shell — no HTTP-level UA gating.
• Prod cloud runs engineVersion 0.18.3 (real Chromium + Chrome 150 UA)
  and still shows "unsupported".

Actual cause

Client-side bot / headless detection (think navigator.webdriver, headless
flags, TLS/JA3 fingerprint, etc.) — plus the report itself loads asynchronously
("collecting data from thousands of sources…"). This is a different, harder
problem class than a UA string, and a real headless Chrome with a correct UA is
blocked all the same.

Direction (not done yet)

A proper stealth / anti-detect tier: harden the existing STEALTH_JS
(webdriver/plugins masking), pair with residential proxy egress, and handle
async-rendered reports. No promises on fully-protected sites like CARFAX, but
this is the right track. Tracking scope + progress here.

Thanks @itsSteven_Phan for the detailed repro 🙏

[us]

update on this one, with numbers.

the block on www.carfax.ca is an azure front door 403, and the engine already classifies it as a hard block and fires the residential recovery arm. the arm was losing on the clock, not on detection: it ran with a 12 s budget and a residential chrome render of that page measures 12.3 to 16.3 s end to end. so it finished on 2 of 11 attempts. with the residential tier pinned and enough budget the same page cleared 6 of 6 times with the full 7.7k of markdown.

PR #510 raises the arm budget to 20 s and couples the residential tier's navigation budget to it. only requests the ladder already failed pay the extra time.

what is still open: the specific vehicle history report url in the original report carries an id token we do not have, so the async report render itself is unverified. keeping this open until that is measured.

[us]

this is on production now. www.carfax.ca clears through the residential tier on the automatic path: 4 of 4 attempts today, each returning the full page (about 7.7k characters of markdown) in roughly 15 to 17 s, where the same requests failed 9 of 11 times before #510. the response still reports the front door's 403 in metadata.statusCode on a cleared page; that is cosmetic and tracked separately.

the specific report url from the original message needs an id we do not have, so if a vehicle history report page still comes back blocked, reopen with the url and we will take it from there. thanks @itsSteven_Phan for the repro that started this.