policy-crds.md

Design: Policy CRDs

Status: implemented · API group: inferencecache.io/v1alpha1

This document tracks the policy-side CRDs that sit beside CacheBackend. These resources are the declarative policy and observability surface; routing and serving decisions remain in the gRPC server and runtime adapters.

CachePolicy

CachePolicy is namespaced. It controls cache lookup and eviction behavior.

Field	Type	Purpose
spec.eviction	enum	Index eviction algorithm applied when the index exceeds its entry cap. LRU (default) evicts the oldest-by-lastSeen entry first; LFU evicts the lowest-access-count entry first, breaking ties on the oldest lastSeen. Access counts do not age — the evictionTTL sweep removes stale entries regardless of algorithm, so LFU does not pin hot-but-stale entries. The controller lower-cases this and propagates it on ResolvedPolicy. The index reads it on the cap-based sweep (to order victims) and, in LFU namespaces, on the lookup path (to record which entries a delivered LookupRoute hint credits — a timed-out lookup credits nothing); it never changes a lookup result, and the TTL sweep is algorithm-independent.
spec.evictionTTL	duration	Maximum usable lifetime for cache entries.
spec.minimumPrefixTokens	integer	Minimum requested prefix token count. Enforces "tiny prompts don't surface PREFIX_MATCH or TENANT_HOT" (i.e. no positive cache-evidence hint). With affinityRouting: Disabled the gate fires as a pre-lookup short-circuit straight to NO_HINT (cheap path: no index touch). With affinityRouting: Enabled (the kubebuilder default) the request goes through the full lookup so the index can classify UNKNOWN_TENANT / UNKNOWN_MODEL / UNKNOWN_HASH_SCHEME diagnostics before any fallback fires (precedence: diagnostic codes > AFFINITY_HINT); if the ranker would have returned PREFIX_MATCH or TENANT_HOT, the handler downgrades it to StrategyNone as a result-side filter, and the affinity fallback then surfaces AFFINITY_HINT with a stable single-replica pick. Either path enforces the operator intent; the fallback reason code is affinity-toggle-dependent. Minimum 0.
spec.minimumMatchedTokens	integer	Minimum matched prefix token count required AFTER the index lookup runs for PREFIX_MATCH to surface. Replicas whose matched overlap falls below this floor are filtered; if none survive, the response downgrades off the prefix-match path to StrategyNone, which surfaces as AFFINITY_HINT under affinityRouting: Enabled (the kubebuilder default) with a usable seed + serving replica or as NO_HINT under affinityRouting: Disabled. The CRD field has a +kubebuilder:default=64 marker, so the apiserver materializes 64 (4 KV blocks at the typical 16-token block size) on any CR that omits the field. The server independently applies the same value as DefaultMinimumMatchedTokens when a tenant has no CachePolicy at all — so trivial 1-block chat-template overlaps are filtered in both shapes. Set to 0 on the CR to disable enforcement for that namespace (e.g. raw-recall benchmarking); the server-side DefaultMinimumMatchedTokens is the fallback ONLY for tenants without a CR. Distinct from minimumPrefixTokens — that field is a request-side gate; this is a result-side floor. Minimum 0.
spec.routingFloorScore	stringified float (e.g. "0.1", "5", "0")	Per-replica score below which a PREFIX_MATCH response is downgraded off the prefix-match path. Applied AFTER the distinguishing-power-aware ranker computes scores. Overlaps held by every replica (chat-template framing, RAG corpus headers, custom system prompts) see distinguishing_power = 0, score = 0, and this floor catches them. The downgrade lands on StrategyNone, which surfaces as AFFINITY_HINT under affinityRouting: Enabled (the default) with a usable seed + serving replica or as NO_HINT under affinityRouting: Disabled. The CRD has a +kubebuilder:default="0.1" marker so the apiserver materializes "0.1" on any CR that omits the field; the server independently applies the same value (as DefaultRoutingFloorScore) when a tenant has no CachePolicy at all. Set to "0" on the CR to disable enforcement for that namespace (raw-recall benchmarking). Composes with minimumMatchedTokens — the matched-tokens floor is applied first per-replica, then this score floor gates the top survivor's score. Both floors can downgrade independently; an operator can disable either by setting it to its opt-out value. Distinct from minimumPrefixTokens — that's a request-side gate; this is a result-side floor on the realized score. Pattern-validated at admission.
spec.lookupTimeoutMs	integer	Lookup latency budget in milliseconds. Minimum 0.
spec.strategy.enableChainMatching	boolean	Enables the block-hash chain matcher for LookupRoute requests that carry block_hashes + block_token_counts. Default true preserves the longest-common-leading-run behavior. When false, the handler strips chain fields before lookup and uses the legacy exact prefix_hash path.
spec.strategy.requireChain	boolean	Requires callers to provide a valid block-hash chain before the index is touched. Default false keeps legacy exact-prefix clients working. When true and a request has no chain, the server returns empty scores with reason_code: POLICY_REQUIRES_CHAIN (fail-open to normal gateway routing). Admission rejects requireChain: true with enableChainMatching: false.
spec.strategy.enableTenantHot	boolean	Allows the TENANT_HOT soft locality fallback. Default true preserves current behavior. When false, a tenant-hot result is downgraded to NO_HINT while prefix matches and diagnostic misses still behave normally.
spec.affinityRouting	enum Enabled | Disabled	Toggles the consistent-hash fallback on the LookupRoute StrategyNone branch. With Enabled (the +kubebuilder:default), a prompt that would otherwise land on NO_HINT AND has all the preconditions met (well-formed contract keys: non-empty tenant_id / model_id / hash_scheme; well-formed chain arrays — both balanced or both empty; a usable seed — non-empty block_hashes OR non-empty prefix_hash; AND at least one replica in servingByScope[(tenant, model, hash_scheme)]) is mod'd against the index-known SCHEME-AWARE replica set for (tenant, model, hash_scheme) — read from the same servingByScope accelerator the TENANT_HOT path uses, so a vLLM request can never pin to an SGLang replica — and a single stable replica is returned with reason_code: AFFINITY_HINT. Cases that fail the preconditions (no seed, no serving replica, empty contract keys, malformed chains, cold-start globally-empty index) stay on NO_HINT even with affinity Enabled. Repeat prompts pin to the same replica and warm T1 on diffuse single-turn workloads (chatbot, RAG with distinct corpus chunks per request). With Disabled, NO_HINT is returned as before and the gateway round-robins; useful for raw-recall benchmarking and ranker debugging where the goal is to see the prefix-match path's unaided behavior. The diagnostic codes UNKNOWN_TENANT / UNKNOWN_MODEL / UNKNOWN_HASH_SCHEME and TIMEOUT keep precedence over AFFINITY_HINT; affinity never preempts a real PREFIX_MATCH or TENANT_HOT that cleared the request-side gates (one exception: a tiny request below the per-namespace minimumPrefixTokens gate has its positive-hint result — including TENANT_HOT — downgraded to StrategyNone, so the affinity fallback can still fire on it; the operator intent "tiny prompts don’t surface a positive hint" outranks the TENANT_HOT-vs-affinity precedence). Affinity bypasses minimumPrefixTokens (stable replica pinning is essentially free even on short prompts) by routing those small requests through the full lookup instead of the pre-lookup short-circuit, so the diagnostic codes still surface for wrong-scheme/wrong-tenant misconfigurations even on tiny prompts. A request with empty hash_scheme or a chain whose two parallel arrays disagree in length stays on NO_HINT (structurally malformed input shouldn't be papered over). See grpc-contract.md § "Affinity routing" for the full semantics.

status.conditions and status.observedGeneration are reserved for controller observations.

Runtime propagation (controller → server /policy) is described in policy-propagation.md: evictionTTL drives per-tenant index eviction; minimumPrefixTokens and lookupTimeoutMs are enforced on the LookupRoute path before and around the index call respectively (when affinityRouting: Disabled, the minimumPrefixTokens pre-lookup short-circuit elides the index call entirely; when affinityRouting: Enabled — the default — the gate still applies but the request runs the full lookup so the index can classify UNKNOWN_* diagnostics before the affinity fallback fires); minimumMatchedTokens is enforced on the LookupRoute path after the index returns (downgrades sub-floor matches off the prefix-match path to StrategyNone, which surfaces as AFFINITY_HINT under default-enabled affinity or NO_HINT when disabled); routingFloorScore is enforced on the LookupRoute path after the distinguishing-power-aware ranker scores each candidate (downgrades whole responses whose top score falls below the floor, with the same AFFINITY_HINT vs NO_HINT split as the matched-tokens row); affinityRouting activates on the StrategyNone branch after both floors have run, returning AFFINITY_HINT instead of NO_HINT when enabled; strategy gates which lookup strategies may surface; and eviction selects the per-namespace cap-based eviction algorithm. The two eviction knobs are orthogonal: evictionTTL removes stale entries on the freshness sweep regardless of algorithm, while eviction only decides which entries the cap sweep drops when the index is over its entry cap. The three lookup-filtering knobs are orthogonal too: minimumPrefixTokens bounds the request, minimumMatchedTokens bounds the realized matched-tokens count per replica, and routingFloorScore bounds the realized per-replica score; affinityRouting is the post-filter fallback that decides whether the otherwise-NO_HINT response carries an AFFINITY_HINT stable replica.

CacheTenant

CacheTenant is namespaced. It defines tenant identity and quota.

Field	Type	Purpose
spec.tenantID	string	Required non-empty external tenant identifier used by gateway and engine traffic. The string inferencecache.io/probe is reserved for the server's functional self-test and is rejected by the validating admission webhook on CREATE (always) and on UPDATE that newly introduces the value (via filterIntroducedErrors). Unchanged legacy CRs predating this rule continue to admit on unrelated edits — the v1alpha1 tightening seam.
spec.quota.maxIndexEntries	integer	Maximum distinct index prefixes attributed to the tenant. Minimum 0. Enforced at ingest (see policy-propagation.md).
spec.isolationMode	enum	Fairness in the current phase.
spec.crypto	object	Reserved for future cryptographic isolation settings.

status.indexEntries, status.conditions, and status.observedGeneration expose observed tenant state.

There is deliberately no spec.quota.maxMemoryBytes or status.memoryUsed. The cache plane only surfaces a max* quota for a resource it authoritatively owns — the index entry table. Engine KV memory is a shared, tenant-unaware pool (vLLM/LMCache key by block hash, not tenant), so the control plane can neither enforce a per-tenant byte budget nor honestly attribute bytes per tenant (ReplicaStats.cache_memory_bytes is the engine total, double-counted across tenants sharing an engine). Per-tenant byte isolation is an engine/runtime concern (separate engine Deployments + pod memory limits).

PromptTemplate

PromptTemplate is namespaced. It gives the rendering layer a template body plus cache-relevant slot declarations.

Field	Type	Purpose
spec.body	string	Required non-empty template text.
spec.slots[]	list	Slot declarations keyed by name.
spec.slots[].name	string	Required non-empty slot identifier used by the template body.
spec.slots[].type	enum	Required Stable or Mutable. Stable slots participate in prefix stability; mutable slots do not.
spec.slots[].required	boolean	Whether callers must provide the slot.
spec.slots[].description	string	Human-readable slot notes.

status.templateRevision is a stable revision identifier for cache invalidation.

PDTopology

PDTopology is namespaced. It declares prefill/decode pools and accelerator classes for phase-disaggregated serving. The reconciler is future work; the type is shipped now so clients can reference the contract.

Field	Type	Purpose
spec.prefillPools[]	list	Prefill pools keyed by name.
spec.decodePools[]	list	Decode pools keyed by name.
spec.acceleratorTypes[]	list	Accelerator classes keyed by name.
*.name	string	Required non-empty identifier for each pool or accelerator type.
prefillPools[].matchLabels, decodePools[].matchLabels	map	Selects pods or nodes for the pool.
prefillPools[].replicas, decodePools[].replicas	integer	Desired pool size. Minimum 0.
prefillPools[].acceleratorType, decodePools[].acceleratorType	string	References spec.acceleratorTypes[].name.
acceleratorTypes[].vendor, acceleratorTypes[].model	string	Descriptive accelerator metadata.
acceleratorTypes[].matchLabels	map	Labels identifying matching nodes or pods.

status.conditions and status.observedGeneration are reserved for future topology reconciliation.

CacheIndex

CacheIndex is cluster-scoped and status-only. It reflects the server's in-memory cache aggregate for observability (kubectl get cacheindex); it is not a routing substrate.

• The CRD has no user-configurable spec. For v1alpha1 compatibility, it accepts an omitted spec or the legacy empty spec: {} shape, but it does not define writable spec fields.
• The controller owns the singleton object and writes only the status subresource.
• Status carries replica, tenant, and prefix summaries. It never stores KV tensors or prompt text.
• status.replicas[] is a map-list keyed on id (the v1alpha1 surface; unchanged for backward compatibility). Each row carries an optional tenant field for source disambiguation; the controller publishes only replicas that have reported stats, so the id key is unique in practice. If two stats-reporting replicas sharing a pod name across namespaces ever collide on id in a single tick, the controller picks the lexicographically-later tenant deterministically and the tenant field on the published row identifies which one was chosen.
• The status is populated by the controller scraping the server's internal /snapshot endpoint. /snapshot.replicas[] is keyed internally by (tenant, replicaId) (so prefix-only replicas with no stats remain attributable per-namespace) and carries per-replica prefixCount and lastEventAt, which the controller projects into CacheBackend.status.indexParticipation. Prefix-only replicas are deliberately omitted from CacheIndex.status.replicas[] — that surface is for replicas with reported stats only.
• status.tenants[].memoryUsed is deprecated and always 0. Per-tenant memory is not honestly attributable on a shared, tenant-unaware engine (ReplicaStats.cache_memory_bytes is the engine total, double-counted across tenants sharing it — the same enforcement-boundary reasoning that removed CacheTenant.status.memoryUsed, see crd-contract.md). The controller hard-zeroes it (never copies the snapshot value), and the field is intentionally not omitempty so it always serializes as memoryUsed: 0, keeping the published v1alpha1 shape stable for clients; it is retained for wire/shape compatibility and scheduled for removal at v1beta1. For a real memory signal read the per-replica engine total at status.replicas[].cacheMemoryBytes.
• Pointer status-field convention — hitRate (on both status.replicas[] and status.tenants[]) is *string and status.tenants[].indexEntries is *int64, so a nil/omitted key is distinct from an observed "0"/0. hitRate nil means "not yet reported" (the stats reporter has not emitted); the controller emits it only when the snapshot's presence bit is set (or, for an older server that predates the bit, when a non-zero lastUpdate/hitRate proves stats were reported). indexEntries nil means "not yet computed" (no snapshot observed) — but the poller runs only after a successful scrape, so an emitted tenant row always carries a real count (a present 0 is a genuinely empty tenant). This matches the per-instance surfaces (CacheBackend.status.indexParticipation.hitRate and CacheTenant.status.indexEntries) so the two read consistently. See ../concepts/cacheindex-cluster-aggregate.md.