Skip to content

[Fix] Readmit PIC addition requests when segment-state capacity is unavailable - #8

Open
morluto wants to merge 5 commits into
redai-studio:mainfrom
morluto:agent/pic-allocation-fallback-pr
Open

[Fix] Readmit PIC addition requests when segment-state capacity is unavailable#8
morluto wants to merge 5 commits into
redai-studio:mainfrom
morluto:agent/pic-allocation-fallback-pr

Conversation

@morluto

@morluto morluto commented Jul 28, 2026

Copy link
Copy Markdown

Motivation

A request can satisfy the token and KV-cache budgets but still fail PIC admission when every reusable Mamba state slot is locked. The current path raises RuntimeError, aborting the entire batch even though the affected request could be processed through normal full-prompt recomputation.

PIC candidate
     |
     v
state slots available? -- yes --> admit reduced PIC work
     |
     no
     v
evict unlocked entries
     |
     v
enough now? ---------- yes --> admit reduced PIC work
     |
     no
     v
price full recompute against token/KV budgets
     |
     +-- fits --> admit full prompt
     |
     +-- no fit --> return NO_TOKEN

Modifications

  • Account for required Mamba state slots before committing PIC admission.
  • Attempt normal unlocked-entry eviction first.
  • If capacity remains unavailable, reconsider the request using its full prompt.
  • Clear discarded prefix-match metadata so normal admission cannot lock or load back stale cache state.
  • Admit full recomputation when the normal token and KV budgets permit it.
  • Return NO_TOKEN when neither PIC admission nor full recomputation fits.
  • Centralize the calculation of Mamba slots required by a new request.

This is scoped to non-RoPE PIC addition mode. Transition-mode behavior is unchanged.

Accuracy Tests

The fallback changes scheduling and admission decisions; it does not change model state calculations.

Speed Tests and Profiling

No performance claim is made. The fallback is entered only when PIC state-slot allocation cannot satisfy the request.

Test Plan

16 passed

Coverage includes:

  • Normal PIC admission when slots are available.
  • Recovery after evicting unlocked entries.
  • Full-prompt readmission when all candidate entries are locked.
  • Reset of discarded generic prefix-match and protection metadata.
  • NO_TOKEN when full recomputation also exceeds the available budget.
  • Preservation of existing non-addition behavior.

Closes #4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] PIC addition aborts a batch when segment-state allocation cannot evict

1 participant