Skip to content

fix(deviceplugin): combine kubelet resource device blocks - #745

Merged
thxCode merged 1 commit into
mainfrom
fix/kubelet-device-blocks
Oct 8, 2026
Merged

thxCode merged 1 commit into
mainfrom
fix/kubelet-device-blocks

Conversation

@thxCode

@thxCode thxCode commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind bug
/area devicemanager

What this PR does / why we need it:

Kubelet can report a two-card container as two ContainerDevices blocks for the same resource. Allocation verification compared each one-card block with the complete Pod annotation. This falsely reported an identity conflict and blocked subsequent allocations on the node.

Combine device IDs from all blocks of the same container and resource before verification. Keep annotation comparison and unknown-device refusal unchanged.

Which issue(s) this PR links to:

NONE

Special notes for your reviewer:

  • Regression tests cover complete, split, reversed, and empty blocks, plus missing and unknown cards.
  • The split-block test fails on the previous implementation with the observed identity conflict.
  • Device-plugin tests, race tests, make lint, and the full make test suite passed. The full suite ran in a clean checkout of this commit.
  • Live regression on an eight-card NVIDIA node: the fixed operator admitted four two-card ModelDeployment containers. TP=2, DP=2, EP=4 and eight inference requests passed.
  • The DP=2 to DP=4 engine retest did not complete. Ray placement groups were created on the four intended Pods; two engine actors remained pending and inference timed out. Instance logical-slicing cases were prepared but not run.
  • Independent source review found no new defect. The existing set-based comparison of duplicate IDs is unchanged.

Does this PR introduce a user-facing change?

Fix false allocation conflicts when kubelet reports a multi-card container in separate blocks for the same accelerator resource.

@gpustack-code-review

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

@thxCode
thxCode merged commit 79cf904 into main Oct 8, 2026
7 checks passed
@thxCode
thxCode deleted the fix/kubelet-device-blocks branch October 8, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant