[RFC]: Programmatic Session-Aware KV Cache Management

[karen-sy]

Motivation.

Authors: @karen-sy

TL;DR: We propose a narrow, router-initiated hint surface as the primary API for programmatic, session-aware KV management directives.

Introduction

Note

For this RFC, we will define “router” as an orchestrator on top of multiple engine units.

Performant inference on agentic workloads requires session-aware KV block management. To predict the priority value of a given KV cache block in such workloads, we need complementary information from the router and the engine. The router is aware of session lifecycle and reuse patterns across workers; the engine is individually aware of the KV cache status it holds. Hence, for vLLM to execute a maximally informed cache policy, (1) the router should express policy intent, and (2) vLLM should resolve that intent against exact cache state and execute it.

Currently, KV cache lifecycle is managed naively, mostly via LRU:

request arrives
  -> router chooses target worker via KV overlap/load
  -> target worker checks local cache
       hit:  reuse
       miss: recompute or apply local offload policy

This is insufficient for multiple scenarios, such as:

• another worker already holds the prefix;
• the workload knows a request will resume soon;
• a session has ended and its KV should be demoted/freed;
• the orchestrator wants to protect high-value KV;
• or local policy cannot tell a short tool gap from a long one.

The missing abstraction is an observable surface where the orchestrator biases the cache manager without owning scheduler or memory internals. To this end, we propose a narrow, router-initiated hint surface (”KvHint”) which enables an external router to pass KV cache intent to vLLM. This avoids invasive changes to the existing engine scheduler and cache manager: vLLM keeps ownership of scheduling and memory and is free to clip, defer, reject, or accept any KvHint.

KvHints communicate intent for a broad range of KV management policies. Some are session-scoped (e.g. “Evict all blocks for Session-123”), while others (e.g. “Retain a single specific block with hash 0x1234”) do not require session identity. Especially for policies which require long-term (spanning session lifecycle) KV management, we intend to leverage G2+ cache as a shared retention and movement substrate.

flowchart LR

R["Router policy"] --> K["Provider-neutral KvHint"]

K --> E["vLLM engine<br/>(scheduler, cache manager, KV connectors)"]

E <--> G1["G1: HBM KV"]

E <--> G2["G2: host KV"]

E -. "Phase 3" .-> X["Shared G2 or G3+ storage<br/>(e.g. Mooncake)"]

Loading

Preliminaries: vLLM KV Cache

vLLM’s prefix cache separates logical prefix identity from physical cache allocation. Full reusable token blocks are represented by chained BlockHash values:

hA   = hash(ROOT, A, extra_keys)
hAB  = hash(hA,   B, extra_keys)
hABC = hash(hAB,  C, extra_keys)

The existing logical-to-physical path is:

+-------------------+   add group_id   +-------------------------+   key in   +-----------------------+   resolves   +-------------------+
| BlockHash         | ---------------> | BlockHashWithGroupId    | ---------> | BlockHashToBlockMap   | ----------> | KVCacheBlock      |
| - logical prefix  |                  | - group-specific key    |            | - logical lookup     |             | - physical ID     |
| - chained parent  |                  | - hash + cache group ID |            | - duplicate copies   |             | - reference count |
+-------------------+                  +-------------------------+            +-----------------------+             +-------------------+

We propose associating session membership and in-session position with the logical BlockHash layer through a separate SessionPrefixIndex:

KvHint(selector = session S)
        |
        v
SessionPrefixIndex
    session S -> {hA, hAB, hABC}          # logical BlockHash values
        |
        +-- G1: add cache group ID
        |       hAB -> (hAB, group 0)
        |             = BlockHashWithGroupId
        |                 |
        |                 v
        |          BlockHashToBlockMap
        |                 |
        |                 v
        |          KVCacheBlock copy/copies
        |
        +-- G2: pass the logical keys to the connector,
                which resolves its connector-owned copies

High Level Roadmap

This is a three-phase plan. Phase 1 introduces engine-local session-to-KV resolution. Phase 2 defines the KvHint selector and action surface and implements bounded in-engine execution. Phase 3 integrates router policy, shared storage, and external orchestration frameworks such as Dynamo.

At the end, the programmatic KV management should be a bidirectional process between the engine and control plane:

Engine -> control plane:
  association, residency, eviction and transfer events
  accepted / clipped / deferred / rejected / missing outcome

Control plane -> engine:
  bounded KvHints/directives

Phase 1: Session-addressable KV cache

Requirements: Tie session associate/membership information with vLLM KV blocks. The abstraction should support a many-to-many association:

one session -> many branches/block extents
one logical block -> many sessions
one logical key -> potentially multiple physical copies

Proposed design: first-class index of KV blocks’ session membership and in-session position.

We propose an option (enabled via e.g. --enable-session-to-block-indexing) to maintain a persistent worker-local index of "which block does each session own", such that vLLM can resolve, in-engine, the block identities targeted by session-level directives.

flowchart TB
    subgraph O["Orchestrator"]
        direction LR
        R["Request session_id"] --> S["Session-relative policy"] --> K["session_id<br/>optional BlockHash boundary"]
    end

    subgraph V["vLLM engine"]
        direction LR
        H["Request BlockHash chain"] --> I["SessionPrefixIndex"] --> G["BlockHashWithGroupId"]
        G --> M["G1 map / G2 connector lookup"] --> P["Physical KV copies"]
    end

    O -->|"KvHint {hint_type, target_session: session_id}"| V

Loading

The SessionPrefixIndex stores logical topology using BlockHash. Cache-group IDs are added only when resolving selected logical nodes to physical copies.

SessionPrefixIndex (--enable-session-prefix-index)
                              ROOT
                            /      \
                          hA        hX
                       /   |   \      \
                     hAB  hAD  hAE    hXY
                      |                 |
                    hABC              hXYZ

Parent links:
  parent[hABC] = hAB; parent[hAB]  = hA; parent[hAD]  = hA, ...
Session frontiers:
  session_1 -> {hABC}, session_2 -> {hAD}, ...
Concurrent branches:
  session_5 -> {hABC, hAD}

Tasks:

1a. Add first-class, request-level session_ids to vLLM requests metadata. Completed in #48048
1b. Evaluate the need for / implement the other proposed PR's in #48049, e.g. update Mooncake / KV connector consumers to read request.session_id.
2. Implement SessionPrefixIndex. Store BlockHash -> parent BlockHash links and session_id -> frontier BlockHash set. Shared ancestors are stored once, while multiple frontiers represent concurrent branches. Optional per-node reference counts support pruning and shared-session accounting.
3. Maintain the index synchronously with cache activity. Maintain the index synchronously during prefix lookup, full-block admission, and decode-block completion. Cache hits must establish session membership even though they produce no BlockStored event. Removing a G1 copy must preserve the extent while a G2 or duplicate copy remains. When the last local copy disappears, move affected frontiers to their nearest locally resident ancestor or remove the branch; retain parent-only nodes while they connect resident descendants.
4. Validate correctness and performance. Validate the indexer behavior and overhead.

Alternative design: Orchestrator externally manages KV blocks’ session membership and in-session position

EDIT TO ADD 8/14: See Dynamo proposal ai-dynamo/dynamo#13279 for an example of orchestrator-driven KV block-session management.
Alternatively, vLLM can avoid maintaining this index by assuming the orchestrator handles that responsibility. In this case, the orchestrator maintains the session-to-block index, and every KvHint it sends to vLLM is expected to provide the exact logical KV block hashes that the hint targets.

flowchart TB

subgraph O["Orchestrator"]

direction LR

R["Global session/radix index"] --> S["Resolve session_id"] --> B["Resolve block hashes for session_id"]

end

subgraph V["vLLM engine"]

direction LR

G["Add cache-group IDs"] --> M["G1 map / G2 connector lookup"]

M --> P["Validate current copies"] --> E["Execute directive"]

end

O -->|"KvHint {hint_type, target_blocks: [list,of,blocks],...}"| V

Loading

Advantages of alternative design:

• vLLM avoids maintaining a duplicate session-to-logical-block index and its memory/update cost.
  • The orchestrator already maintains these associations to evaluate session-aware policy.

Disadvantages of alternative design:

• vLLM lacks a first-class abstraction for addressing “block(s) associated with session_id.”
  • An externally resolved list[blocks] is an orchestrator-side snapshot; this may become stale as blocks are admitted, removed, or moved before or during execution, causing correctness issues.

Phase 2: KvHint-driven KV cache management

Requirements: Build vLLM-side plumbing to consume external KVHints, and translate the hints’ semantics into in-engine KV block management procedures.

Tasks:

1. Define and carry a provider-neutral KvHint . Propagate the envelope from an external orchestrator through request preprocessing and the engine request path to the scheduler, cache manager, and relevant KV connectors.
2. Implement the semantics of each hint carried by KvHint. Resolve the hint’s target blocks via SessionPrefixIndex or exact block hashes, validate the engine’s actual (G1/G2) KV states, execute (or defer/reject) the hint’s cache action accordingly, and report the outcome.

We define the semantics of various KvHints in the “Hint Taxonomy (conceptual)” section.

Phase 3: Integration with Orchestrator Frameworks (e.g. Dynamo)

Requirements: Show end-to-end programmatic KV management: an orchestrator framework emits a policy decision as a KvHint → vLLM consumes the KvHint and performs cache management correctly based on engine state.

Tasks:

1. Dynamo is actively developing router-based KvHints as part of the KVCC library (DEP: KV Cache Controller ai-dynamo/dynamo#11673). We plan to achieve API surface compatibility, such that Dynamo router’s KvHints may be elegantly consumed by / lowered to vLLM’s KvHint APIs developed in the previous two Phases.

Hint Taxonomy (conceptual)

Our design of hints is aligned with: SGLang #27574

EDIT to add 8/22: We are actively working on KvHint development in vLLM. Please see:

1. [RFC]: First-Class, Orchestrator-Agnostic KV Hint Envelope for Agentic Workloads #53421 for initial KvHint protocol standardization and definition within vLLM
2. More coming soon

A KvHint combines:

target:
  which logical KV blocks are affected

action:
  Share | Prefetch | Demote | Pin | Retain

bounds:
  optional TTL, priority, destination, or resource limit

e.g. KvHint{session: sess-123, action: Prefetch, priority: 10, destination: worker-A} 

vLLM resolves the target against SessionPrefixIndex or exact block keys, applies the action to current G1/G2 copies, and reports an accepted, clipped, deferred, rejected, or missing outcome.

flowchart LR

subgraph W1["Worker A"]

direction LR

G1A["G1: HBM KV"] -->|"Demote"| G2A["G2: host KV"]

G2A -->|"Prefetch"| G1A

end

subgraph W2["Worker B"]

direction LR

G1B["G1: HBM KV"] -->|"Demote"| G2B["G2: host KV"]

G2B -->|"Prefetch"| G1B

end

W1 -->|"Share selected KV"| W2

W1 -.->|"Pin / Retain"| W1

W2 -.->|"Pin / Retain"| W2

Loading

Retain

Bias eviction order without guaranteeing protection. A retain hint assigns a relative priority, optionally for a bounded duration; under pressure, vLLM prefers to evict lower-priority KV first. Retained KV may still be evicted when necessary.
There is prior draft work on this: #37003

Share

Make KV available on another worker without recomputation. The orchestrator selects the source, destination, and target KV; vLLM and its connectors validate current source copies, perform the transfer, and publish destination visibility only after completion. Sharing does not necessarily remove the source copy.

Prefetch

Move or copy KV into G1 before it is expected to be used. The common local operation promotes KV from G2 host memory to G1 HBM. If the target is already resident in G1, the hint may complete as a no-op.

Demote

Move KV to G2 instead of discarding it. vLLM first ensures that a valid G2 copy exists, then allows the G1 copy to be reclaimed. Transfer completion and G1 reclamation are separate operations.

Pin

Protect high-value KV from ordinary eviction for a bounded TTL. Pin is not permanent and does not necessarily require G1 residency: vLLM may preserve the target in G2 while allowing its G1 copy to be reclaimed. The engine may clip or reject pins that exceed configured limits.

Design Principles

1. Orchestrator owns policy; engine executes. The agent-graph / workflow intelligence lives outside. The engine understands priority, TTL, session membership, tier - nothing about why.
2. Zero overhead when unused. Un-hinted workloads behave exactly like today. In other words, we are not replacing local prefix matching or LRU principles, but rather augmenting the scope of its capacities.
3. Hints are soft, bounded, and safe to reject. The engine may accept, clip, defer, or ignore. Every hint is observable. Nothing a client says can pin memory unboundedly or deadlock the scheduler.
4. Router-initiated by default. Workloads can still emit intent, but the router is where workload context merges with global KV placement, worker load, health, and admission. In production environments, the router has: a global KV index from events, built-in HA/fault-tolerance, existing overlap/load routing + admission control, and (with the harness<->orchestrator work) trajectory awareness, not just request awareness.
5. API surface is not finalized here: We would like to open discussion for what goes into a KvHint { retention: [{prefix_tokens, ttl_seconds}] } .

[cr-zhao]

Hi @karen-sy, I’d be interested in taking Implementing SessionPrefixIndex in phase 1 if no one is already working on it. I was thinking of starting with the standalone core index abstraction and unit tests, and leaving the cache lifecycle wiring for a follow-up. Would that be ok with you?

[franciscojavierarceo]

this is closely aligned with the architecture we’ve been converging on across agentic-api, llm-d, and vllm. we just opened a draft RFC applying this model to server-managed prompt-cache state for messages: vllm-project/agentic-api#195

we strongly agree with the core boundary here: the orchestrator owns policy and trusted logical identity, while vllm resolves that intent against current physical KV state and executes, clips, or rejects it. in an llm-d deployment, agentic-api derives the session and continuation coordinates after resolving API-level state, then llm-d owns worker selection and its global KV index.

this seems complementary to the concrete agent_hint proposal in #52113 and the typed session/continuation work in #48501. we should align these into one contract where session identity, lifecycle intent, and exact block identity remain distinct.

our controlled responses routing benchmark saw mean KV hit rate increase from 49.54% to 99.08%, with mean continuation latency dropping from 811 ms to about 315 ms. the benchmark is small and directional, but it supports validating this control-plane and engine split under broader agentic workloads.