[Proposal] Unified routing surface

[ryan-lempka]

Proposal co-authored by @ayushag-nv

Background

Today in Switchyard every routing algorithm ends by naming a model and every algorithm carries its own copy of the model catalog.

That suits a route running one strategy against a fixed pair of models. At wider scope it introduces these issues: the same two models are configured four different ways, strategies cannot be combined because each terminates on a model, and selection cannot weigh price or preferences because neither is available where the decision happens.

To successfully unify the existing algorithms there also needs to be a layer that makes a final decision when signals diverge. For instance stage tool signals say X, task classification says Y, and the advisor says Z all at the same time.

In addition, right now switchyard requires customers to categorize models (e.g. efficient/capable) which means they have to make a judgement call on what those definitions mean. Or use an existing configuration we provide that is "blessed". This proposal is a stepping stone (not the final state) towards being able to support an arbitrary pool of models without a need to categorize them explicitly and solves the immediate need of unifying the algorithms.

flowchart LR
  subgraph T["Today: every algorithm holds the catalog"]
    direction LR
    R1[Request] --> A1["Classifier<br/><i>knows sol, luna</i>"]
    R1 --> A2["Stage<br/><i>knows sol, luna</i>"]
    R1 --> A3["Advisor<br/><i>knows sol, luna</i>"]
    A1 --> M1[ModelId]
    A2 --> M1
    A3 --> M1
    M1 --> U1[Upstream]
  end

  subgraph P["Proposed: one place holds the catalog"]
    direction LR
    R2[Request] --> S["Signals<br/><i>task, tool loop, review</i>"]
    S -->|"needs + confidence"| O["Pool optimizer<br/><i>catalog, cost,<br/>budget, preferences</i>"]
    O --> M2[ModelId]
    M2 --> U2[Upstream]
  end

Loading

Goals

One vocabulary for declaring models. Strategies that combine. A deployment that works from a list of models. Less surface area in the algorithm layer.

Non-Goals

• Infrastructure signals in the pool optimizer
• Automatic discovery of model capability from providers
• A learned or online-optimizing policy

Proposal

Algorithms emit what a request needs. A pool optimizer decides which model provides it.

Three core changes:

1. What an algorithm returns

Today. The capability classifier computes an estimate, compares it to a threshold, and throws the estimate away:

let target = if verdict.p_solve >= threshold { &self.efficient } else { &self.capable };
Classification::Scores(vec![Score { confidence: 1.0, target: target.clone() }])

Proposed. The estimate is the output:

Classification::Scores(vec![
    Score { confidence: verdict.p_solve, needs: vec!["efficient".into()] },
])

Other algorithms do the same with their own score.

2. What a Score points at

Today:

struct Score { confidence: f64, target: ModelId }

Proposed:

struct Score { confidence: f64, needs: Vec<String> }

ModelId is why every algorithm needs the catalog today.

3. What a deployment looks like

Today. The same two models, four times, four vocabularies:

strong_target = "sol"          # llm_classifier
weak_target   = "luna"

capable_target   = "sol"       # stage_router
efficient_target = "luna"

advisor_target  = "sol"        # advisor
executor_target = "luna"

targets = ["sol", "luna"]      # random

Proposed. Declare the pool once:

[pool]
models = [
  { target = "sol",  provides = ["capable"] },
  { target = "luna", provides = ["efficient"] },
]
default = "capable"

[routes.switchyard]
id   = "switchyard"
type = "auto"

The above design can be extended to more than two categories (e.g. beyond capable/efficient) and a model can provide multiple categories.

Strategies become configurable signals on the route:

[routes.switchyard.signals]
task      = { judge = "terra", trigger = "user_turn", threshold = 0.9 }
tool_loop = { enabled = true, confidence = 0.5 }
review    = { judge = "sol", max_reviews = 3, confirmations = 2 }

auto replaces the existing route types.

The optimizer

The optimizer takes signals and picks the model. How it settles a disagreement is an open question. Today each algorithm applies its own threshold and selects the top score. This provides a layer that sees every score and can introduce other constraints like cost. Can be extended to take in other preferences from the user.

Proposed.

fn select(scores: &[Score], pool: &PoolContext) -> Result<ModelId>;

When more than one model satisfies the need, prefer one the session has already used, since the provider's cache is warm for that prefix.

Composite routing, before and after

Composite routing first use case allowed LLM classifier to set the tier that stage defaults to when the confidence signal is uncertain. Composite needed its own route type, a processor that wrote into the stage algorithms session state, and a runtime API to set and clear that tier.

Today:

[routes.switchyard]
id   = "switchyard"
type = "composite"

[routes.switchyard.classifier]
target           = "terra"
base_threshold   = 0.9
classify_trigger = "user_turn"

[routes.switchyard.stage]
capable_target       = "sol"
efficient_target     = "luna"
confidence_threshold = 0.5

Proposed:

[routes.switchyard.signals]
task      = { judge = "terra", trigger = "user_turn", threshold = 0.9 }
tool_loop = { enabled = true, confidence = 0.5 }

The two signals are declared side by side and the optimizer resolves them. No route type for the pair, no writing into another algorithm's state.

The same shape covers the pattern where stage calls the classifier itself when its own signals are indecisive:

task = { judge = "terra", trigger = "on_demand", threshold = 0.9 }

What this removes: composite as a route type, the processor that sets the fall-open tier, the runtime setter and the session state behind it, and the capable_target and efficient_target pair inside stage. What replaces it is a pool default for the case where no signal is confident:

[pool]
models = [
  { target = "sol",  provides = ["capable"] },
  { target = "luna", provides = ["efficient"] },
]
default = "capable"

User Experience Impact

After this change a working deployment requires a list of models.

Public API. The TOML schema is the affected surface. [pool], type = "auto", and [routes.*.signals] are added. Existing route types and their keys are eventually replaced. [llm_clients.*], [targets.*], route id, and every HTTP endpoint are unchanged. Sequencing and migration come after the shape is agreed.

Docs. The seven routing-algorithm pages collapse to one for the route type and one for the pool.