Skip to content

feat: Multiple pools per guard - #22

Merged
mkhazraee merged 16 commits into
ai-dynamo:mainfrom
karya0:kapila/multiple-pools-per-guard
Sep 10, 2026
Merged

feat: Multiple pools per guard#22
mkhazraee merged 16 commits into
ai-dynamo:mainfrom
karya0:kapila/multiple-pools-per-guard

Conversation

@karya0

@karya0 karya0 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator
  • Adds end-to-end support for multiple Local DRAM pools per Guard.
  • Each Guard owns one allocation containing independently sized pools,
    while keys may span a variable ordered set of pool-qualified slots

@karya0
karya0 force-pushed the kapila/multiple-pools-per-guard branch 4 times, most recently from f5519bc to 7e9fc77 Compare September 9, 2026 03:43
BREAKING CHANGE: local DRAM residencies and transfer results carry ordered pool-and-slot locations.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
BREAKING CHANGE: Guard claims and grants describe one jointly owned group of pool regions.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
@karya0
karya0 force-pushed the kapila/multiple-pools-per-guard branch from 7e9fc77 to 2a0fa68 Compare September 9, 2026 04:14
Signed-off-by: Kapil Arya <kapila@nvidia.com>
BREAKING CHANGE: capacity_needed_callback now receives an ordered list of pool name and slot count pairs.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
@karya0
karya0 marked this pull request as ready for review September 9, 2026 16:57
Copilot AI lite review requested due to automatic review settings September 9, 2026 16:57
@karya0 karya0 changed the title Multiple pools per guard feat: Multiple pools per guard Sep 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces an API-breaking bug in KVCRCore.fetch() defaulting expected_layout to [""] (failing for non-empty single-pool configs) and has at least one recovery decoding error-path that can escape as an uncaught msgspec.ValidationError.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends KVCR’s Guard/service and recovery pipeline from a single-pool model to supporting multiple pools per Guard-owned allocation, including multi-descriptor block layouts, pool-aware recovery wire format, and updated capacity-pressure semantics.

Changes:

  • Introduces ordered pool descriptors on the Guard protocol wire and binds recovery handback terms to ordered pool geometry/identity.
  • Updates local DRAM residency/eviction to manage grouped allocations across multiple pools and multi-descriptor layouts per key.
  • Refactors and expands unit/integration tests and documentation to match the new pool-group semantics.
File summaries
File Description
tests/unit/test_recovery_mirror.py Updates recovery wire round-trip tests for pool-aware G2 locations and mirror construction.
tests/unit/test_recovery_journal.py Extends snapshot/terms tests to include ordered pool descriptors and allocation identity.
tests/unit/test_kvcr.py Adjusts core/guard startup tests and residency assertions for multi-pool and G3 constraints.
tests/unit/test_kvcr_service.py Reworks service/registry claim tests to validate grouped allocations and pool descriptors.
tests/unit/test_kvcr_service_workflow.py Updates end-to-end daemon workflow tests for pool naming and page-aligned sizing.
tests/unit/test_kvcr_remote_source.py Updates remote FW DRAM wire shape for multi-descriptor destinations and better malformed logging checks.
tests/unit/test_kvcr_local_dram.py Adds multi-descriptor and multi-pool residency/eviction/capacity-pressure behavior tests.
tests/unit/test_guard.py Refactors Guard tests to use new configure/handback APIs and pool-group behavior.
tests/unit/test_guard_protocol.py Adds pool descriptor wire-contract validation and claim/grant layout checks.
tests/unit/test_guard_integration.py Adds integration coverage for two-pool group failover and reclaim behavior.
tests/unit/test_g3.py Adds regression test around G3/full behavior with local slot reuse semantics.
tests/unit/_kvcr_test_utils.py Updates test helpers for nested descriptor lists and capacity callback plumbing.
src/kvcr/remote_fw_dram.py Changes start_write payload decoding and internal ops to support per-key descriptor lists.
src/kvcr/recovery_journal.py Makes recovery wire G2 carry pool-name+slot pairs; binds handback terms to pool descriptors.
src/kvcr/policy_runtime.py Extends eviction queue to track weights (slot counts) and total reclaimable weight.
src/kvcr/local_dram.py Implements multi-pool local DRAM management, multi-descriptor layouts, and per-pool capacity pressure.
src/kvcr/local_disk.py Extracts G3 slot-geometry validation and adapts store path to list-based sources.
src/kvcr/kvcr_service.py Updates service/registry semantics and grant shape for grouped pool allocations.
src/kvcr/guard.py Refactors Guard recovery state to configure pool descriptors atomically and hand back without layout args.
src/kvcr/guard_protocol.py Adds _PoolDescriptor to the wire and validates grant pool layout/offset/size contract.
src/kvcr/core.py Enables multi-pool layouts, multi-descriptor normalization, and per-pool capacity pressure signaling.
src/kvcr/api.py Updates capacity callback type annotation to list of (pool_name, num_slots).
docs/dev-guide.md Updates service daemon documentation for Guard-owned pool groups and ordered pool layout terms.
docs/design_overview.md Updates API contract docs for per-pool capacity pressure requests and multi-pool layouts.
Review details
  • Files reviewed: 24/24 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/kvcr/recovery_journal.py
Comment thread src/kvcr/core.py Outdated
Comment thread src/kvcr/core.py Outdated
Comment thread src/kvcr/local_disk.py Outdated
Comment thread src/kvcr/policy_runtime.py Outdated
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Restore the base constructor spelling to keep the multi-pool diff focused.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
Restore base snapshot-term ordering and journal formatting; retain
malformed-frame coverage while sharing repeated setup.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
Cover native Guard transfers and reclaim with multiple descriptors.
Consolidate overlapping transfer, malformed-input, and capacity cases.

Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapila@nvidia.com>
@mkhazraee
mkhazraee merged commit a053257 into ai-dynamo:main Sep 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants