Routing service

[madison-evans]

the following PR is a re submission of a previously closed PR

Description

This PR adds a Router micro-service to GenAIComps.
The router decides which downstream LLM endpoint is best suited for an incoming prompt and returns that endpoint URL.
It is stateless and supports two interchangeable controller back-ends:

Controller	Technique	Config file	Typical use-case
RouteLLM	Matrix-factorisation ranking trained on preference datasets (e.g. gpt4_judge_battles)	deployment/docker_compose/configs/routellm_config.yaml	Fine-grained complexity routing
Semantic-Router	Embedding-similarity / threshold	deployment/docker_compose/configs/semantic_router_config.yaml	Simple “weak vs strong” intent separation

How the configs fit together

comps/router/
├─ deployment/docker\_compose/configs/
│  ├─ router.yaml                # *Global* — maps strong/weak endpoints & points to per-controller YAMLs
│  ├─ routellm\_config.yaml       # RouteLLM-specific knobs (datasets, MF checkpoints, thresholds)
│  └─ semantic\_router\_config.yaml# Semantic-Router routes & encoder names
└─ src/opea\_router\_microservice.py
• Reads CONFIG\_PATH (env var) → loads router.yaml
• Uses “controller\_config\_paths” section to locate the chosen sub-config
• Instantiates controller via `ControllerFactory`

At runtime, docker compose mounts ./configs → /app/configs in the container.
CONFIG_PATH=/app/configs/router.yaml tells the service where to start.

Deployment:

Docker Compose bundle (deployment/docker_compose/compose.yaml + deploy_router.sh)

---

Issues

n/a — new component

---

Type of change

• New feature (non-breaking change which adds new functionality)
• Bug fix
• Breaking change
• Others (enhancement, documentation, validation, etc.)

---

Dependencies

New PyPI packages added to comps/router/src/requirements.txt:

• routellm — an Intel fork of the RouteLLM project (https://github.com/SAPD-Intel/RouteLLM)
• semantic-router — embedding-based router
• huggingface-hub (pulled transitively)

These depend on the existing stack (FastAPI, Pydantic, etc.).
Runtime requires HF_TOKEN and OPENAI_API_KEY secrets.

---

Tests

End-to-end validation script tests/router/test_router_routellm_on_xeon.sh

1. Builds the router image (comps/router/src/Dockerfile)
2. Spins up the service via Docker Compose
3. Sends two sample prompts
   expects "weak" route for easy math, "strong"
4. Shuts everything down

This assumes CI pipeline already exposes HF_TOKEN and OPENAI_API_KEY, so the script is invoked automatically.

# local smoke-test
export HF_TOKEN=***   OPENAI_API_KEY=***
chmod +x tests/router/test_router_routellm_on_xeon.sh
tests/router/test_router_routellm_on_xeon.sh

[lvliang-intel]

@madison-evans,
please fix the CI issue, you need to add your new service to docker build yaml.

[ftian1]

https://github.com/SAPD-Intel/RouteLLM

have you went through security process for this fork repo to be referenced?

[joshuayao]

Hi @madison-evans Could you please check the CI failures?

[madison-evans]

@madison-evans, please fix the CI issue, you need to add your new service to docker build yaml.

for clarity, are you saying that I need to add my compose.yaml to .github/workflows/docker/compose as router-compose.yaml? Do I have that correct?

[madison-evans]

example-test seems to be failing...

line 17, in <module>
      from llama_index.llms.openai import OpenAI
    File "/usr/local/lib/python3.11/site-packages/llama_index/llms/openai/__init__.py", line 2, in <module>
      from llama_index.llms.openai.responses import OpenAIResponses
    File "/usr/local/lib/python3.11/site-packages/llama_index/llms/openai/responses.py", line 6, in <module>
      from openai.types.responses import (
  ImportError: cannot import name 'ResponseTextAnnotationDeltaEvent' from 'openai.types.responses' (/usr/local/lib/python3.11/site-packages/openai/types/responses/__init__.py)
  + exit 1
  Error: Process completed with exit code 1.

A bit confused why that is. The commits I've made have been contained within comps/router

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/proto/api_protocol.py	92.08% <100.00%> (+0.03%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[chensuyue]

example-test seems to be failing...

line 17, in <module>
      from llama_index.llms.openai import OpenAI
    File "/usr/local/lib/python3.11/site-packages/llama_index/llms/openai/__init__.py", line 2, in <module>
      from llama_index.llms.openai.responses import OpenAIResponses
    File "/usr/local/lib/python3.11/site-packages/llama_index/llms/openai/responses.py", line 6, in <module>
      from openai.types.responses import (
  ImportError: cannot import name 'ResponseTextAnnotationDeltaEvent' from 'openai.types.responses' (/usr/local/lib/python3.11/site-packages/openai/types/responses/__init__.py)
  + exit 1
  Error: Process completed with exit code 1.

A bit confused why that is. The commits I've made have been contained within comps/router

This issue caused by deps update, and has been fixed yesterday. This example-test triggered due to your code update of comps/cores/ folder.

[madison-evans]

all tests are passing now. Ready for review

[haim-barad]

Fixed as per your review comments - @ashahba

[ashahba]

LGTM!

[haim-barad]

What step (I assume last step) is needed for merge? Is "Update Branch" enough? If yes, which method?