fix(crawler): re-validate redirect hops and strip credentials cross-origin

[namann5]

Description

crawl_target() in backend/secuscan/crawler.py followed HTTP redirects automatically (httpx.AsyncClient(follow_redirects=True, max_redirects=10, ...)) with no re-validation of any hop, and it forwarded operator-supplied credentials — including the decrypted vault Authorization: Basic ... header injected by the executor (executor.py:_hydrate_inputs_with_execution_context) — plus session cookies to every redirect destination.

This PR hardens redirect handling in the crawler:

1. SSRF via redirect is closed. Redirect hops are re-validated against the network policy engine before they are fetched, so a hostile or compromised seed cannot pivot the crawler into cloud-metadata (169.254.169.254), loopback, private/CGNAT, or IPv6 link-local/ULA ranges that were never authorized. This matches how the seed target is already validated by _enforce_guardrails, and is active when enforce_network_policy is enabled (the default). Validation is fail-closed: if the policy engine errors, the redirect is blocked.

2. Credential exfiltration via cross-origin redirect is stopped. Credentials (extra_headers such as Authorization, plus cookies) are only sent to the seed origin. On any cross-origin redirect (different scheme, host, or effective port), they are stripped — mirroring browser behavior. This prevents an attacker who controls a redirect from an authorized target from harvesting the operator's decrypted vault credentials at their own listener.

Redirects are now followed manually (follow_redirects=False) to preserve the existing max_redirects and max_size constraints while adding per-hop checks.

Related Issues

Fixes #2369

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

How Has This Been Tested?

• Added testing/backend/unit/test_crawler_redirect_security.py (16 tests) covering:
  • _is_same_origin semantics (scheme / host / port, case-insensitivity, malformed-port fail-closed behavior).
  • _validate_redirect_target blocking cloud-metadata, unsupported schemes, missing hostnames, and failing closed on engine errors.
  • crawl_target aborting a redirect into 169.254.169.254 before the destination is fetched (asserts the metadata endpoint is never requested).
  • crawl_target stripping Authorization/cookies on a cross-origin redirect while preserving them on a same-origin redirect.
  • The manual redirect loop still enforces max_redirects (raises httpx.TooManyRedirects).
• Ran ruff check backend testing/backend — passes.
• Ran the existing crawler-related suites (test_crawler_limits.py, test_crawler_helpers.py, test_crawler_surface_parsing.py, test_tls_verification.py, test_crawler_plugin.py) — all pass except one pre-existing, unrelated metadata assertion (test_crawler_target_field_requires_http_url) that also fails on main and is not executed by CI.
• Full testing/backend/unit suite: 2311 passed; the 26 local failures are pre-existing environment-specific ones (parser sandbox subprocess / ICMP / scheduler timeout) that fail identically on main in this Windows/Python 3.10 environment and are unrelated to this change.

Checklist

• My code follows the code style of this project.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6f7aa95067

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[utksh1]

Request changes: redirect validation checks the hostname through check_access, but the follow-up HTTP request resolves the hostname again. That leaves a DNS-rebinding window between policy validation and the actual fetch. Resolve and pin the address for every redirect hop, or use a transport that connects to the validated address while preserving the Host header; add a rebinding regression test.

[namann5]

@utksh1 Please review

[namann5]

@utksh1 All admin review comments have been addressed and the branch is rebased onto the latest main (including the undici override fix), so all required checks are green and the PR is mergeable. Requesting re-review.

Summary of changes in this PR:

• DNS rebinding fix: the crawler resolves the redirect target's DNS once and pins the validated IP for each redirect hop while preserving the Host header, preventing SSRF redirect and credential-exfiltration.
• Added a rebinding regression test plus tests for DNS resolution, validated-IP return, resolution failure, and IP-pinning behavior.
• Honoring settings.network_policy_failure_mode == log_only (logs instead of raising) for redirect denials, addressing the P2 comment.
• Rebased on current main; CI: backend lint/tests/unit, frontend checks, formatting-hygiene all pass.