[RFC]: Session-Aware KV Cache Hints for Agentic Workloads

[FermatGo]

Motivation.

Agentic applications often know more about future KV cache reuse than the inference engine:

• a child agent is waiting for a tool and will probably resume soon;
• several agents share a long system/tool prefix;
• a branch has completed and its KV cache can be released;
• an offloaded branch will be resumed and can be prefetched.

Today, after a request finishes, these prefixes are ordinary free cached blocks managed by the same replacement policy. vLLM cannot distinguish a temporarily idle but valuable agent context from a prefix that will never be used again.

This RFC proposes optional agent_hint metadata so an application can communicate session identity, a short retention TTL, and explicit cache management operations. The goal is to improve prefix-cache hit rate and TTFT for multi-turn and multi-agent workloads.

The proposal addresses an information gap rather than replacing vLLM's cache policy. An agent runtime knows when a tool call is outstanding, a sub-agent has finished, a conversation has been suspended, or a checkpoint is about to be resumed. vLLM knows which KV blocks are resident and when capacity is needed. Neither side can make the best decision with only its own information. Hints provide a narrow, best-effort contract between those two layers.

Workload scenarios and expected value

The largest opportunity is not generic chat traffic, but workloads with long prefixes and explicit lifecycle transitions:

Application event	Hint/action	Expected system value
A long-running agent waits for a tool or user response	Protect the request's cache for a short TTL, or offload it for a longer pause	Avoid an accidental LRU eviction and repeated prefill when the agent resumes
A leader creates multiple workers from a common context	Associate parent and child sessions; protect the active branches	Preserve normal content-based sharing of the common prefix while retaining each branch only as long as it is useful
A child agent completes or a failed branch is abandoned	Evict that session's cached context	Reclaim capacity earlier than an engine-only timeout or replacement policy can
An application switches, resumes, or rewinds a session	Prefetch the target context before inference	Overlap KV movement with application state transition and reduce resume TTFT
Context is compacted or archived	Evict obsolete cache, or offload context that may be restored	Align physical KV residency with the application's actual context lifecycle
A session moves between instances in pooled/disaggregated serving	Retain or prefetch the corresponding remote KV entries	Improve cross-instance reuse and avoid recomputing a long prefix after routing changes

These actions target three concrete outcomes:

• Lower and more stable TTFT: fewer long-prefix cache misses and an opportunity to hide remote-load latency behind tool execution, scheduling, or session switching.
• Higher effective KV capacity: dead branches can be released promptly, while temporarily idle but expensive contexts receive bounded protection. The same HBM budget can therefore support a more useful working set, which may translate into more concurrent agent contexts or less capacity needed for the same service target.
• Less recomputation and data movement: the application can distinguish a likely resume from a terminal state, allowing vLLM to spend prefill compute and connector bandwidth on contexts with known reuse value.

The exact gain depends on prefix length, pause duration, cache pressure, and backend bandwidth. This RFC therefore treats these as benchmark hypotheses, not guaranteed improvements. The rollout section proposes measuring hit rate, recomputed prompt tokens, TTFT, throughput, and transfer volume.

This is a cache-policy hint, not a new cache identity:

• block hashes and prefix-cache matching remain unchanged;
• identical content can still be reused across sessions;
• session_id is not included in the cache key;
• vLLM may ignore or override hints under memory pressure;
• requests without hints behave exactly as they do today.

Goals

• Retain likely-to-be-reused cache blocks for a bounded time.
• Release known-dead session context promptly.
• Allow optional offload, prefetch, and evict operations.
• Keep session policy separate from BlockPool.
• Add negligible overhead to requests that do not use the feature.
• Support local cache first and optional CPU/remote KV backends later.
• Let applications express useful lifecycle intent without exposing vLLM block size, block tables, or physical block IDs.

Proposed Change.

1. Initial request-level implementation

The complete design can describe subranges, but the implementation currently being validated intentionally starts with a simpler profile: protection, offload, prefetch, and eviction operate on a whole request context. The agent runtime does not calculate or send block indices.

For a normal inference request, vLLM associates session_id with the complete cacheable prefix materialized for that request. If cache_control is present, that complete association receives the requested TTL. For a pure management request, the operation applies to the complete context currently recorded for session_id.

class AgentHintParams(OpenAIBaseModel):
    session_id: str | None = None
    parent_session_id: str | None = None
    cache_control: CacheControlParams | None = None
    context_management: ContextManagementParams | None = None


class CacheControlParams(OpenAIBaseModel):
    type: Literal["ephemeral"] = "ephemeral"
    ttl: float = Field(default=300.0, ge=0, le=3600)
    # Optional fine-grained extension. None protects the whole request context.
    block_offset: int | None = Field(default=None, ge=0)


class ContextManagementParams(OpenAIBaseModel):
    manage_request: bool = False
    edits: list[ContextEdit] = Field(default_factory=list)


class ContextEdit(OpenAIBaseModel):
    type: Literal["offload", "prefetch", "evict"]
    # Optional fine-grained extension. Both None target the whole request context.
    block_start: int | None = Field(default=None, ge=0)
    block_end: int | None = Field(default=None, ge=0)  # exclusive

Protect a request context:

{
  "model": "example-model",
  "messages": [{"role": "user", "content": "Inspect the repository."}],
  "agent_hint": {
    "session_id": "worker-7",
    "parent_session_id": "orchestrator-1",
    "cache_control": {"type": "ephemeral", "ttl": 300}
  }
}

Prefetch a recorded session before it resumes:

{
  "model": "example-model",
  "messages": [{"role": "user", "content": ""}],
  "agent_hint": {
    "session_id": "worker-7",
    "context_management": {
      "manage_request": true,
      "edits": [{"type": "prefetch"}]
    }
  }
}

This initial request-level implementation has several practical advantages for a vLLM integration:

• the public API does not expose allocator-specific block indices;
• all mapping work occurs at existing request allocation/completion boundaries;
• validation can focus on lifecycle semantics and measurable benefit before adding range-selection complexity;
• the implementation remains useful for the common cases of suspend, resume, branch completion, and session deletion.

Internally vLLM still operates on blocks, but that is an implementation detail: SAM resolves the recorded request/session association to the current physical blocks and validates the expected hashes before changing metadata. In other words, the application is request-aware while vLLM remains block-aware.

2. Keep fine-grained fields, but defer their use in the initial implementation

block_offset, block_start, and block_end remain in the request model as optional fields. They preserve the complete design and provide a compatible path to fine-grained management after the request-level workflow is validated. The current tests simply omit them:

• cache_control.block_offset=None protects the whole cacheable request context;
• ContextEdit.block_start=None and block_end=None target the whole recorded request context;
• when fine-grained support is enabled, block_start and block_end must be supplied together and describe a session-relative half-open logical range;
• a supplied block_offset protects the cacheable blocks from start to that logical offset of the request.

These values are logical indices, never physical block IDs. Therefore retaining the fields does not require the simplified client workflow to understand vLLM's allocator or physical block table.

When context_management.manage_request is true, vLLM skips model execution and applies the requested edits. With the default value false, edits run after the normal request completes.

In the actual running process, message to token to block index translation is implemented by the upper-layer routing, or the parsing can be placed in the VLLM.

3. Add a small SessionAwareManager

Add a SessionAwareManager (SAM) beside KVCacheManager.

SAM owns:

• session and optional parent/child relationships;
• session-to-block and block-to-session indexes;
• block-level TTL entries;
• validation and execution of context-management edits.

BlockPool continues to own physical block allocation, the free queue, and the prefix hash map. It does not store session IDs or session trees.

flowchart LR
    API["Request + agent_hint"] --> Scheduler["V1 Scheduler"]
    Scheduler --> KVCM["KVCacheManager"]
    Scheduler --> SAM["SessionAwareManager"]
    KVCM --> Pool["Existing BlockPool"]
    SAM -->|"one metadata API"| KVCM
    SAM -. "optional lifecycle operations" .-> Connector["KV connector capability"]
    Connector -.-> Backend["CPU / remote KV store"]

Loading

SAM records a logical block index and expected block hash with each association.

4. Add two aggregate fields to KVCacheBlock

@dataclass(slots=True)
class KVCacheBlock:
    # Existing fields remain unchanged.
    _session_ref_cnt: int = 0
    _ttl_expire_at: float = 0.0

_session_ref_cnt is independent of the existing active-request ref_cnt. Detailed session IDs stay in SAM, so the additional per-block memory is constant and prefix-cache lookup does not need a session parameter.

5. Add one KVCacheManager metadata API

SAM does not modify blocks directly. It uses one guarded method:

def update_block_meta(
    self,
    *,
    block_id: int,
    expected_block_hash: BlockHashWithGroupId | None = None,
    session_ref_delta: int = 0,
    ttl_expire_at: float | None = None,
) -> BlockMetaUpdateResult:
    ...

The method validates that:

• the physical block still contains the expected cached content;
• _session_ref_cnt cannot become negative;
• SAM never changes active-request ref_cnt;
• null blocks are not retained.

If the block is free, the method also asks the free queue to reconsider its allocation priority.

KVCacheManager only needs to notify SAM at existing request boundaries:

1. after cached/new blocks are assigned to a request;
2. when a request completes or a block is reassigned.

6. Best-effort TTL retention

Free blocks are logically classified as:

Class	Condition	Allocation order
A	TTL expired and no session reference	First
B	TTL expired and has session references	Second
C	TTL not expired	Last resort

This can be implemented by extending the existing free linked list with two boundaries; it does not require a new cache allocator.

[A: ordinary free] -> [B: session-associated] -> [C: TTL-protected]

TTL is deliberately soft:

1. allocate A blocks first;
2. then allocate B blocks;
3. if only C blocks remain and inference needs memory, break a C reservation.

This guarantees that inaccurate hints cannot block scheduling or cause an avoidable out-of-memory failure.

TTLManager and timer wheel

SAM owns a TTLManager backed by a fixed-size timer wheel. Entries are keyed by (block_id, session_id):

@dataclass
class TTLBlockEntry:
    block_id: int
    session_id: str
    expire_at: float

register() is invoked when a block with a TTL is allocated or the TTL of a block is updated, and notifies the remote backend to keep the block alive of the corresponding session. remove() cancels an entry during evict or session cleanup, and notifies the remote backend to stop the block keepalive for the corresponding session. On every scheduler iteration, tick(now) advances elapsed wheel slots and calls SAM.on_ttl_expired(block_id, session_id) for truly expired entries. on_ttl_expired will call remove to delete the entry.

The callback removes the session record, decrements the aggregate session reference, recomputes the block's latest remaining deadline, and calls update_block_meta(). Thus, expiration of one session does not remove a later TTL owned by another session.

With a one-second tick and the proposed one-hour maximum TTL, the wheel has 3,600 slots. Registration, refresh, and removal are O(1); each tick processes only entries in elapsed slots. time.monotonic() is used throughout. Allocation also checks the block deadline, so scheduler delay or a skipped tick cannot prevent expired C-zone blocks from being reclaimed.

7. Session lifecycle and management operations

Session registration is lazy and idempotent. Reusing a session refreshes its associations without double-incrementing _session_ref_cnt.

free_session(session_id) removes that session's records and TTL entries. free_session_tree(session_id) optionally releases children before their parent. Neither operation invalidates blocks used by active requests or other sessions.

In the initial implementation, management edits resolve to all blocks in the recorded request context for the target session. A later fine-grained extension may use session-relative half-open logical block ranges:

• evict: cancel selected local and remote retention.
• offload: persist selected KV data remotely, then release its local retention.
• prefetch: load selected remote KV data into the local cache.

Active request references always win. An edit never invalidates a block while its existing ref_cnt is positive.

Offload and prefetch are capability based. If no backend supports an operation, vLLM returns unsupported; the local prefix-cache behavior remains valid.

8. Pooling and remote KV stores

Local cache management is the first milestone. Pooled CPU or remote KV stores need the same lifecycle semantics, but the integration should not make vLLM core understand backend-specific PoolKeys, transfer threads, or eviction policies.

Compatibility path for current connectors

For connectors that only expose their existing lookup/save/load paths, an optional SessionAwarePoolingManager (SPM) can act as an adapter. It listens to SAM lifecycle events and maintains a temporary mapping:

session_id -> block hashes / connector keys

After a connector finishes saving KV data, SPM associates the saved keys with the session. It can then:

• refresh active or TTL-protected keys at low frequency when the backend's exists operation also refreshes LRU state;
• stop keep-alive after a session or key becomes orphaned, allowing the backend's existing LRU policy to reclaim it;
• submit a bounded, low-priority load through the existing connector for prefetch, while reserving local capacity for normal inference;
• confirm that offload/save completed before local retention is released.

This adapter provides a way to test end-to-end semantics without requiring an immediate change to every connector or backend. It is deliberately optional and is not involved in local block allocation or prefix matching.

flowchart LR
    Edit["offload / prefetch / evict"] --> SAM["SAM: resolve logical blocks"]
    SAM --> Local["KVCacheManager: local metadata"]
    SAM --> SPM["SPM: resolve PoolKeys"]
    SPM --> Tracker["SessionKeyTracker"]
    SPM --> Queue["Bounded prefetch / eviction queue"]
    Queue --> Store["CPU or remote KV store"]

Loading

9. Expected vLLM change surface

The initial request-level implementation keeps the required vLLM core change small and localized:

Area	Change
Request protocol/internal request	Add optional request-level agent_hint fields
New SAM module	Session indexes, TTL, edit controller
KVCacheBlock	Add two aggregate metadata fields
KVCacheManager	Add one metadata method and lifecycle notifications
Free-block queue	Add metadata-based priority/reclassification
Scheduler	Construct SAM, tick TTL, dispatch management requests

Fine-grained ranges and pooled-cache management are separate, optional layers:

Optional area	Change
Logical range support	Translate stable logical ranges only if the community chooses to expose them
Compatibility SPM	PoolKey tracker, keep-alive, and remote operation queues for current connectors
Connector capability	Propagate logical session/request identity and expose remote observation/control
Remote backend	No required change; precise TTL/delete remains optional

The proposal does not change:

• block hash calculation;
• prefix-cache matching;
• cache keys or cache_salt;
• attention kernels;
• model runners;
• block tables sent to workers;
• sampling or generated results;
• existing connector behavior when no session-aware capability is enabled.

SAM, range support, and pooled-cache integration should have separate experimental flags. With them disabled, the current cache and connector paths remain unchanged. In particular, connector support is not a prerequisite for reviewing or merging the local request-level path.

Feedback Period.

No response

CC List.

@JuneHM @fc-liu @luokui183 @wangxiaochao6 @LookAround0301

Any Other Things.

Co-Authors

@HiC4Sh1e @JiahongZhang-Work @Xavier-Zeng @socrahow @liudi60

Results

Based on the SWE-bench Verified data set and Qwen3-235B model, openJiuwen did a series of comparative tests: covering typical scenarios such as Bug fixes, feature development, and code refactoring, simulating 10 users concurrently using JiuwenSwarm to perform tasks, comparing the effect with and without computing affinity. The results show that the first token delay (TTFT) is reduced by 57.46% , the model request end-to-end (E2E) delay is reduced by 27.61% , the Prefix Cache hit rate is increased by 33% , and the peak Cache usage is reduced by 25.24% .

Additionally, we also conducted a simple test without the agent framework. Experimental configuration: 20 sessions, 5 concurrent requests, request length 10-30K. By sending requests directly to vLLM. We observed that the TTFT was optimized by approximately 21%.

Metrics	Agent Hint OFF	Agent Hint ON	Agent Hint ON + Evict
TTFT (average)	4.52 s	3.58 s	3.57 s
TTFT (P50)	4.57 s	3.47 s	3.57 s
TPOT (average)	0.030 s	0.034 s	0.032 s
TPOT (P50)	0.029 s	0.034 s	0.032 s

Backward compatibility and safety

Requests without agent_hint behave exactly as before, and hints cannot affect generated tokens. session_id is not authorization: management operations must be tenant-scoped, while cache_salt continues to provide cache isolation. TTL, sessions, records, edit ranges, and prefetch work must be bounded; session IDs must not be metric labels.

Performance and observability

The target overhead is two scalars per block, O(1) reclassification, and work proportional to associated blocks at request boundaries. There is no session lookup during prefix matching or per-token decode work. Remote tracking and control run only when both a connector and session hints are enabled.

Suggested metrics include:

• sessions, records, protected blocks, and pressure overrides;
• edit/prefetch results by operation and status;
• tracked/shared/orphaned keys, keep-alive, and remote eviction counts;
• connector transfer bytes, queue delay, completion latency, and failures;
• prefix-cache hit rate and TTFT for hinted versus unhinted requests.

Rollout and testing

The change can be split into small PRs:

1. request schema and shadow SAM accounting;
2. local TTL retention and session cleanup;
3. local evict and structured management responses;
4. SPM with PoolKey tracking and keep-alive;
5. optional connector offload/prefetch/explicit-evict capabilities.

Benchmarks should compare prefix-cache hit rate, recomputed prompt tokens, TTFT, throughput, scheduler CPU time, and metadata memory overhead against the current LRU baseline.

Future direction

Planned direct connector integration

The longer-term integration should make session-aware management an optional connector capability instead of relying on SPM to infer remote state from side-effects. A capable connector is the component that already knows how local block hashes map to remote keys, whether data is resident, and when an asynchronous transfer has completed. It is therefore the right boundary for both observing and controlling remote KV state.

The exact interface should be developed with connector maintainers, but the capability would cover operations equivalent to:

class SupportsSessionAwareKVManagement(Protocol):
    def register_cache(self, session_id, request_id, block_hashes) -> None: ...
    def offload(self, session_id, request_id=None) -> OperationHandle: ...
    def prefetch(self, session_id, request_id=None) -> OperationHandle: ...
    def evict(self, session_id, request_id=None) -> OperationHandle: ...
    def release(self, session_id, request_id=None) -> None: ...
    def query_residency(self, session_id, request_id=None) -> Residency: ...

The names and argument types above are illustrative, not a proposed final API. The important properties are:

• direct visibility: connectors report save/load completion, remote residency, misses, and backend pressure through structured results or events;
• direct control: offload, prefetch, retain/release, and explicit eviction use connector methods when supported, rather than scheduler-created virtual requests or assumed exists() side-effects;
• logical inputs: core passes a session/request association and content hashes; connector-specific keys remain private to the connector;
• capability negotiation: unsupported operations return unsupported and never change normal request execution;
• backend freedom: a connector may implement exact TTL/delete, map release to stopped keep-alive, or rely on backend LRU according to its capabilities.

flowchart LR
    Hint["request-level hint"] --> SAM["SAM: logical lifecycle"]
    SAM --> Local["KVCacheManager: local policy"]
    SAM --> API["optional connector capability"]
    API --> Observe["residency and completion events"]
    API --> Control["offload / prefetch / evict / release"]
    Observe --> Store["pooled CPU or remote KV store"]
    Control --> Store

Loading

AscendStore can be an initial integration by propagating optional session_id and request identity in connector metadata, reporting saved block hashes after put completion, and reusing its lookup/save/load paths. As connector capability methods mature, its adapter-specific tracking can move behind the connector boundary. Other KV connectors can opt in independently; none of these remote capabilities are required for the initial local implementation.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[franciscojavierarceo]

this is great, we’re pretty aligned with the proposal. we just opened a draft RFC for server-managed prompt-cache state for messages in agentic-api: vllm-project/agentic-api#195

the split we’re proposing is that agentic-api normalizes protocol-specific state into trusted session coordinates and lifecycle intent, llm-d owns cluster-level placement, and vllm owns physical KV resolution and execution. clients never manage engine cache pointers, and session identity remains separate from exact token and block identity.

this also looks complementary to the provider-neutral KvHint work in #51428 and the typed identity work in #48501. it would be useful to align agent_hint and KvHint into one lifecycle contract, while keeping Anthropic cache_control as protocol-level intent that the control plane lowers into that contract.

we’ve seen strong directional results from the same architecture. in a controlled 12-pair responses benchmark, KV-aware routing improved mean KV hit rate from 49.54% to 99.08% and reduced mean continuation latency from 811 ms to about 315 ms. it’s not directly comparable to your agent benchmark, but both results support the same opportunity.