feat(router): add load-aware generate paths

[bvolpato]

Purpose

Extensions can expose inference-generate-compatible endpoints beyond /inference/v1/generate. Add extra_generate_paths and --extra-generate-paths so configured endpoints use existing typed routing instead of transparent fallback.

Configured paths:

• validate with InferenceGenerateRequest
• extract token_ids for cache-aware routing
• forward to same worker path
• use existing retries, circuit breaking, load accounting, authorization, and concurrency limits
• support regular and prefill/decode routers

Built-in handlers and unconfigured fallback behavior remain unchanged. Invalid, duplicate, dynamic, or conflicting paths fail startup validation.

Cancellation-safe global load accounting remains separate in #200. Related consumer update: NovaSky-AI/SkyRL#1988.

Test Plan

• Register /custom/v1/generate against two delayed workers.
• Send overlapping requests and verify cache-aware routing accounts for in-flight load.
• Verify extension JSON fields survive typed forwarding.
• Verify prefill/decode routing clamps nested sampling limits.
• Validate Python and Rust configuration surfaces and reserved-route rejection.

Test Result

• make fmt: passed.
• make check: passed with three existing Clippy warnings.
• cargo test --all-features: passed, including 486 library tests and 48 API endpoint tests.
• uv run --extra dev pytest py_test/unit -q: 109 passed, 4 skipped.
• uvx black --check: passed for touched Python files.
• uvx ruff check: passed for touched Python files.
• git diff --check: passed.

---

Essential Elements of an Effective PR Description Checklist

• Purpose documented.
• Test plan documented.
• Test results documented.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c35714f5e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8d88d153c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".