feat(renderer): let camoufox wait out a JS challenge instead of reporting it

[rqi14]

The gap

run_sequence_inner evaluates outerHTML exactly once, then immediately judges what came back:

https://github.com/us/crw/blob/c74bcf3/crates/crw-renderer/src/camoufox.rs#L223-L247

let tab_id = self.create_tab(url, user_id, session_key, deadline).await?;
let html = self.evaluate_outer_html(&tab_id, user_id, deadline).await?;
...
if let Some(kind) = looks_like_wall(&html) {
    return Err(CrwError::RendererError(format!(
        "camoufox: bot {kind} detected in rendered HTML"
    )));
}

A Cloudflare-style JS challenge resolves client-side, seconds after navigation. That single immediate evaluate can therefore only ever observe the interstitial. The tier reports a retryable wall for pages it would have gotten had it looked again — and since camoufox is normally the last tier, its failure is the request's failure.

Why this is the missing half of an existing pattern

Waiting out a challenge is already a first-class concept here — the CDP tiers do exactly this:

Piece	Location
CHALLENGE_MAX_RETRIES, CHALLENGE_POLL_INTERVAL_MS	cdp.rs:24-28
challenge_retry_budget() — explicit deadline reservation	cdp.rs:30-39
chrome_challenge_max_retries config key	config.rs:711-717
.with_challenge_retries(...) at both CDP sites	lib.rs:879, lib.rs:927

Camoufox — the tier explicitly documented as the one that "covers fingerprint/bot-challenge blocks that the CDP tiers cannot pass" — is the only tier that never got the loop.

This does not overlap the recent wall work (294eb3d, 82a56f6, c74bcf3). Those made an unclearable wall report honestly instead of shipping as a billed success; none of them attempt to clear one. This is the complementary half: don't call it a wall until you've given it the chance the CDP tiers already get.

The change

renderer.camoufox_challenge_wait_ms, #[serde(default)] → 20s. Some(0) restores today's exact single-shot behaviour.

Configured the same way the CDP tiers are — CamoufoxRenderer::new(...).with_challenge_wait(...) mirrors CdpRenderer::new(...).with_challenge_retries(...). new keeps its four arguments, so this is purely additive for any external caller.

The loop polls only looks_like_wall(&html) == Some("challenge"). The "wall" kinds ("attention required! | cloudflare", "enable javascript and cookies to continue") are terminal refusals rather than work in progress — polling those would spend the entire ceiling to arrive at the identical error. A clean page never enters the loop, since looks_like_wall is None on the first evaluate.

Budget interaction — raising this before you have to ask

challenge_budget = self.challenge_wait.min(deadline.remaining()), so a challenge that never clears cannot outlive the request. But note the asymmetry with CDP: challenge_retry_budget() reserves its poll time in the outer timeout, whereas this clamps against whatever remains. When remaining < 20s this tier can therefore consume nearly all of it.

In practice camoufox is the last tier in the ladder (include_in_auto = false by default; when enabled it is pushed last), so there is normally nothing downstream to starve. If you would rather it be explicit, I am happy to add a camoufox analogue of challenge_retry_budget() and reserve it in tier_timeouts_from — say the word and I will push it.

Verification

$ cargo fmt --check                                              # clean
$ cargo clippy -p crw-core -p crw-renderer --all-targets \
      --features crw-renderer/camoufox                           # no warnings in touched files
$ cargo check --workspace --all-targets                          # default features (camoufox off) still builds

$ cargo test -p crw-core
test result: ok. 500 passed; 0 failed        (lib, incl. the 2 new config tests)
test result: ok. 13/2/3/21 passed; 0 failed  (config_tests/api_casing/error_tests/types_tests)

$ cargo test -p crw-renderer --features camoufox --lib camoufox:: -- --test-threads=1
test camoufox::tests::challenge_that_clears_on_a_later_poll_is_returned_as_content ... ok
test camoufox::tests::terminal_wall_is_not_polled ... ok
test camoufox::tests::wall_detection_returns_retryable_renderer_error ... ok
... 15 passed; 0 failed; finished in 0.33s

$ bash scripts/docs-guards.sh && bash scripts/check-doc-links.sh \
    && bash scripts/check-crate-graph-doc.sh && bash scripts/check-cli-command-doc.sh \
    && bash scripts/check-skill-route-links.sh && bash scripts/check-no-process-exit.sh
all PASS

Tests added

• config::tests::renderer_config_default_all_fields — the new field added to the per-field default list
• config::tests::camoufox_challenge_wait_default_and_override — default / override / explicit-0-disables, shaped after camoufox_timeout_default_and_override
• camoufox::tests::challenge_that_clears_on_a_later_poll_is_returned_as_content — the actual new behaviour: interstitial first, real page on the next poll
• camoufox::tests::terminal_wall_is_not_polled — a "wall" fails in well under one poll interval even with a 30s ceiling
• wall_detection_returns_retryable_renderer_error — given a small ceiling so the never-clears path stays sub-second (0.07s) instead of spending the production budget

Docs

config.default.toml and docs/docs/js-rendering.md, alongside the sibling camoufox_timeout_ms. Only the authored Markdown is touched; docs/<slug>/index.html regenerates on main via google-indexing.yml, matching how upstream feature commits do it.

Not reproduced locally

cargo test -p crw-renderer --lib also reports 4 failures in http_only::tests (direct_blackhole_*, connection_failure_catches_connect_timeout, is_retriable_error_false_for_connect_timeout), and cargo test -p crw-server --test api reports 4 more. All eight fail identically on unmodified c74bcf3 on this machine — they depend on 192.0.2.1/198.18.x.x resolving as unroutable, and this network intercepts both ranges. Unrelated to this change.

[us]

Thanks for this, and for raising the budget interaction yourself rather than leaving it to be
found. The idea is right: camoufox is documented as the tier that covers challenges the CDP
tiers cannot pass, and it is the only tier that never got the wait loop. The plumbing is
careful too, and it is worth saying what I checked and found clean before the part that had to
change: cleanup is not regressed (run_sequence binds the inner result and runs
destroy_session at statement level, so the new ? inside the loop still tears the session
down), there is no stale-status problem (run_sequence_inner hardcodes Ok((200, html))
before and after), and config compatibility is fine in both directions since nothing in
config.rs uses deny_unknown_fields.

I have pushed a commit onto your branch. Three changes.

1. The predicate the loop polls on was not fit for looping. looks_like_wall matched the
bare /cdn-cgi/challenge-platform directory, and that marker was deliberately removed from
this codebase twice, each time after a live capture:

• crw-crawl/src/single.rs: "Cloudflare re-injects that telemetry loader into the CLEARED page
  too (measured at byte ~782k of a real post-solve 783k Glassdoor page that carries NO
  _cf_chl_opt)".
• crw-renderer/src/detector.rs: the interstitial is
  /cdn-cgi/challenge-platform/h/g/orchestrate/... while the ordinary Bot-Management response,
  "INCLUDING a cleared page", is /cdn-cgi/challenge-platform/scripts/jsd/main.js. "The /h/
  segment is what separates them." It also records that this is how cloak.rs's accept gate
  came to reject its own successful solves.

looks_like_wall was the third copy of that list and the only one that still carried the bare
directory. Before this PR that cost an instant false rejection. With the poll it would cost the
full ceiling first: camoufox clears the interstitial, the real page still ships
scripts/jsd/main.js, the predicate keeps returning Some("challenge"), the loop polls 20s,
and the post-loop check throws the real content away anyway, on exactly the site class the tier
exists for. Narrowed to challenge-platform/h/.

The fixture in challenge_that_clears_on_a_later_poll_is_returned_as_content now carries the
telemetry loader, which is what a real cleared page looks like. Proof that it bites, by putting
the old marker back on the fixed branch:

old marker: test challenge_that_clears_on_a_later_poll_is_returned_as_content ... FAILED
new marker: test challenge_that_clears_on_a_later_poll_is_returned_as_content ... ok

2. Terminal walls are now matched first. NEEDLES.iter().find(...) returns the first match
in LIST order, not document order, and all five challenge needles preceded both wall needles.
So a page carrying both kinds classified as a clearing challenge and got polled for the whole
ceiling to reach the same refusal, which is the outcome the loop's own comment says it avoids.
Same proof, with the original ordering restored:

original order: left: Some("challenge")   right: Some("wall")   ... FAILED
wall-first:     ... ok

3. The deadline is re-checked between the sleep and the evaluate. The sleep can consume
what was left of the shared deadline, and the next evaluate_outer_html would then return
CrwError::Timeout, replacing this tier's informative wall RendererError (and the antibot
attribution that rides on it) with a bare "timed out", along with a different HTTP status.

Also replaced assert!(started.elapsed() < CHALLENGE_POLL_INTERVAL) with a count of the
evaluate calls the mock actually received. It proves the same thing and cannot flake on a
loaded runner.

cargo test -p crw-renderer --features camoufox is 15/15, clippy with camoufox,cdp and
-D warnings is clean, cargo fmt --check is clean.

Two things still open.

None of this tier's tests run in CI. camoufox is default-off in all three crates and no
workflow passes the feature, so cargo test --workspace never compiles the module. A
cargo test -p crw-renderer --features camoufox step in ci.yml fixes it; I could not push
that here because a workflow file needs a scope this token does not have, so it will be added
separately.

The budget reservation you raised is still worth doing, and it does not belong in
camoufox.rs. challenge_budget bounds accumulated sleep rather than loop wall time, and
the cloak recovery arm that runs after the ladder is gated on
deadline.remaining() >= CLOAK_ARM_FLOOR_MS (24s) unless cloak_recover_on_cf is set, which
defaults to false. A 20s poll can push a configured cloak arm under that floor where today
camoufox failed in about 2s and left it 38s. The right place is the per-tier budget the ladder
already hands out (tier_timeouts_from), the way the CDP tiers reserve theirs, rather than a
constant inside the tier. Happy to take it as a follow-up.

For scope: this tier is opt-in and is not configured on the hosted deployment, so the blast
radius of the whole change is self-hosters running a camoufox sidecar.

[us]

pushed one more commit on top, after a second look at the branch as it would land.

the wall-first ordering was classifying the standard cloudflare managed-challenge interstitial as a terminal wall: "enable javascript and cookies to continue" is the noscript line that page carries, and the repo's own capture in tests/egress_latch_no_latch_on_cf.rs has it next to the orchestrator script. so the poll never ran on the page it exists for. now only the "attention required" title is terminal-first, and the noscript line counts as a refusal only with no challenge marker around it and outside <noscript>.

also in that commit: one failed evaluate per streak is tolerated (the challenge clears by reloading the tab, and an evaluate in that window fails), each evaluate is bounded by what is left of the ceiling, the loop stops at 250 ms of remaining deadline instead of dispatching a call that can only time out, and when a cloak endpoint is configured in auto mode with camoufox in the ladder the poll leaves the cloak arm's floor untouched. the docs and config.default.toml had both camoufox budgets under [renderer.camoufox], where the endpoint table silently ignores them; they now sit in [renderer].

cargo test -p crw-renderer --features cdp,camoufox,cloak is 28 passed, clippy clean for camoufox alone, cloak alone and all three. #509 makes ci run that combination.