Skip to content

Feature/aiv 785 - #2254

Closed
jokerhaos wants to merge 2 commits into
smg-project:mainfrom
astro-web3:feature/aiv-785
Closed

Feature/aiv 785#2254
jokerhaos wants to merge 2 commits into
smg-project:mainfrom
astro-web3:feature/aiv-785

Conversation

@jokerhaos

@jokerhaos jokerhaos commented Aug 21, 2026

Copy link
Copy Markdown

Description

Problem

Solution

Changes

Test Plan

Checklist
  • cargo +nightly fmt passes
  • cargo clippy --all-targets --all-features -- -D warnings passes
  • (Optional) Documentation updated
  • (Optional) Please join us on Slack #sig-smg to discuss, review, and merge PRs

joker added 2 commits August 21, 2026 18:16
joker
feat(router): add cache_aware_length policy with long/short pool split
a13b497 
Add a new self-contained routing policy that splits healthy workers into
a long pool (label pool=long) and a short pool (remaining workers), then
applies cache-affinity routing on top. Designed for both P/D disaggregated
prefill fleets and regular single-node deployments.

Routing pipeline (5 steps):
- Step 1: health filter (empty fleet -> 503)
- Step 2: global imbalance check (same formula as cache_aware)
- Step 3: cache hit via per-model string radix tree
- Step 4: long/short pool split by uncached prefill tokens
  (X-Prompt-Tokens header or char-level estimate, 100K threshold)
- Step 5: record tree + return

Configuration: cache_threshold, balance_abs/rel_threshold,
eviction_interval_secs, max_tree_size, chars_per_token,
long_prefill_threshold, long_pool_max_load, short_pool_max_load.

Wired through PolicyConfig enum, factory, validation, CLI, and registry
init/remove hooks (single + PD paths). Does not touch mesh sync, KV
event monitor, or hash index -- fully additive, zero impact on existing
policies.

Tests: 14 unit tests + 16 E2E tests covering all decision-table rows.
joker
ci: add build-docker-images workflow for GHCR + ZOT + Aliyun registry…
7f7319b 
… sync
@coderabbitai

coderabbitai Bot commented Aug 21, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@jokerhaos, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 6 minutes

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3cba852d-b505-4831-853f-78dbce431da7

📥 Commits

Reviewing files that changed from the base of the PR and between e22c88c and 7f7319b.

📒 Files selected for processing (9)
  • .github/workflows/build-docker-images.yml
  • model_gateway/src/config/types.rs
  • model_gateway/src/config/validation.rs
  • model_gateway/src/main.rs
  • model_gateway/src/policies/cache_aware_length.rs
  • model_gateway/src/policies/factory.rs
  • model_gateway/src/policies/mod.rs
  • model_gateway/src/policies/registry.rs
  • model_gateway/src/routers/http/router.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added ci CI/CD configuration changes model-gateway Model gateway crate changes labels Aug 21, 2026
@jokerhaos jokerhaos closed this Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CatherineSue CatherineSue Awaiting requested review from CatherineSue CatherineSue is a code owner
@key4ng key4ng Awaiting requested review from key4ng key4ng is a code owner
@slin1237 slin1237 Awaiting requested review from slin1237 slin1237 is a code owner
@XinyueZhang369 XinyueZhang369 Awaiting requested review from XinyueZhang369 XinyueZhang369 is a code owner

Assignees

No one assigned

Labels

ci CI/CD configuration changes model-gateway Model gateway crate changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jokerhaos