Repository navigation
feat(preflight): report unhealthy accelerators and skip capability probes - #803
Merged
Merged
Conversation
…obes - add capDeviceHealth and annotate detection detail with unhealthy counts - emit unavailable device-health checks for unhealthy accelerators - filter unhealthy accelerators out of driver, slice, and management checks - add unit tests for detection detail, device-health checks, and filtering Task 1 of preflight-unhealthy-accelerators. Signed-off-by: thxCode <thxcode0824@gmail.com>
…oss all manufacturers - extract evaluateDeviceUnhealthy helpers in amd, hygon, iluvatar, metax, mthreads, nvidia, and thead - add table-driven TestEvaluateDeviceUnhealthy unit tests for each manufacturer - verify alignDeviceGroups retains unhealthy accelerators across all 9 manufacturers Task 2 of preflight-unhealthy-accelerators. Signed-off-by: thxCode <thxcode0824@gmail.com>
Document the device-health capability row, the (<N> unhealthy) detection detail suffix, and kubectl exec inspection using detect and preflight inside a running Device Manager Pod. Task 3 of preflight-unhealthy-accelerators. Signed-off-by: thxCode <thxcode0824@gmail.com>
|
🔍 OpenCodeReview found 1 issue(s) in this PR.
📄
|
…rers Signed-off-by: thxCode <thxcode0824@gmail.com>
Collaborator
Author
Review Feedback Summary
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind enhancement
/area devicemanager
What this PR does / why we need it:
Report
Unhealthyaccelerators (Accelerator.Status.Unhealthy == true) consistently indevice-manager preflightacross all 9 supported manufacturers (nvidia,ascend,amd,hygon,thead,cambricon,iluvatar,metax,mthreads), matching the inventory reported by in-clusterDevicesanddevice-manager detect, and skip downstream capability probes on faulty cards while continuing to preflight healthy cards on the same node.pkg/devicemanager/preflight):detection.state: okand the full accelerator count when a group contains unhealthy cards, while annotating the group fragment indetection.detailas<Name> x<Total> (<Unhealthy> unhealthy).PreflightCheckrow (capability: device-health,mode: unnamed,state: unavailable,depth: declared) for each unhealthy accelerator soFailed()andReport()return a non-zero exit code for bring-up automation.partitionHealthyGroupsso unhealthy cards are excluded from vendorPreflightAcceleratorchecks, preload staging,measureSlicedcontainer probes, andcheckManagementwhile healthy cards in the same group are still preflighted.pkg/devicemanager/detector):evaluateDeviceUnhealthyhealth-evaluation helpers in the remaining 7 vendor detector packages (amd,hygon,iluvatar,metax,mthreads,nvidia,thead) and add table-driven unit tests covering healthy, unhealthy, and unreadable/unsupported cases across all 9 manufacturers.TestAlignDeviceGroups_RetainsUnhealthyAcceleratorsAcrossAllManufacturersto verify that an 8-card group with 1 unhealthy card retains all 8 cards throughalignDeviceGroupswithStatus.Unhealthypreserved on the faulty card.docs/modules/devices/preflight.md,docs/README.md,specs/):device-healthcheck row, the<Name> x<Total> (<Unhealthy> unhealthy)detection detail suffix, and in-Podkubectl execinspection usinggpustack-operator device-manager detectandgpustack-operator device-manager preflight --manufacturer=<manufacturer>.Which issue(s) this PR links to:
NONE
Special notes for your reviewer:
specs/2026-10-29-preflight-unhealthy-accelerators.mdgo test -race ./pkg/devicemanager/preflight/... ./pkg/devicemanager/detector/...),make lint,make lint docs, and packaged containerexec(detect+preflight) runs across AMD (2 × Radeon RX 7800 XT), NVIDIA (2 × GeForce RTX 4090), and Ascend (8 × 910B2).Does this PR introduce a user-facing change?