PAIR routing paths and capability direction

[vonargo]

Love what y'all are doing and appreciate the opportunity to collaborate!

I just did pull request #64 and I think I understand the direction here, but want to check before going further against feature #24.

My read:

• PAIR currently routes a request based mainly on which upstream service has the requested model
• An upstream service might be a raw vLLM server, or something like Dynamo or llm-d managing multiple workers behind one endpoint
• The proposed capability work could let PAIR choose between those upstream services based on fit, not just model name
• PAIR would still route one request to one upstream service
• PAIR would not manage the workers, KV cache, GPU placement, or other inference scheduling behind that service

A few questions:

• Is that the direction you all have in mind for capability routing?
• Should capabilities describe the upstream service, the models it exposes, or both?
• Should PAIR infer requirements from the request, accept them explicitly from the client, or eventually support both?
• Do you see model name remaining the primary selector with capabilities as filters, or could capability fit become the primary selector?
• Is the intended boundary that PAIR chooses the right upstream service and systems like Dynamo or llm-d handle everything behind it?

[Noah-Tervalon-Nvidia]

@NV-sschneider can answer this better, but we are having some discussions about exactly what direction we want to support here in terms of other styles of upstream resources (ex. k8s clusters).
On the idea of routing:

• We do plan to keep the boundary of choosing the right upstream and being a passthrough.
• We plan to update routing to have a handful more options rather than the naive version right now. We are working on these changes right now, but the goal is eventually to support things like kv cache, throughput measurement (effectively how strong is a given machine), queue depth, and ideally measure how large a workload might be and do some prediction, cost, context size, throughput etc. There are a handful of other ideas that @sherief-nv is thinking about.
• model name will be a determining factor for now until we get a model router for requests and can drop it to optional, we have no current stance on developing that as part of this project's scheduler, or using an external model router (or potentially letting users choose a model router). This space is just too young for us to have a defined opinion yet.

[NV-sschneider]

Agree with all the points above.

I think in general, we will want some fuzzier routing, but ultimately be true to the upstream requester. However, the user should be the one to choose how "loose" we want the routing to be. For example, one of the first things i'd love to see is routing across different quants.