Skip to content

feat(preflight): report unhealthy accelerators and skip capability probes - #803

Merged
thxCode merged 4 commits into
mainfrom
preflight-unhealthy
Oct 29, 2026
Merged

thxCode merged 4 commits into
mainfrom
preflight-unhealthy

Conversation

@thxCode

@thxCode thxCode commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind enhancement
/area devicemanager

What this PR does / why we need it:

Report Unhealthy accelerators (Accelerator.Status.Unhealthy == true) consistently in device-manager preflight across all 9 supported manufacturers (nvidia, ascend, amd, hygon, thead, cambricon, iluvatar, metax, mthreads), matching the inventory reported by in-cluster Devices and device-manager detect, and skip downstream capability probes on faulty cards while continuing to preflight healthy cards on the same node.

  • T1 (pkg/devicemanager/preflight):
    • Keep detection.state: ok and the full accelerator count when a group contains unhealthy cards, while annotating the group fragment in detection.detail as <Name> x<Total> (<Unhealthy> unhealthy).
    • Emit a PreflightCheck row (capability: device-health, mode: unnamed, state: unavailable, depth: declared) for each unhealthy accelerator so Failed() and Report() return a non-zero exit code for bring-up automation.
    • Partition each group via partitionHealthyGroups so unhealthy cards are excluded from vendor PreflightAccelerator checks, preload staging, measureSliced container probes, and checkManagement while healthy cards in the same group are still preflighted.
  • T2 (pkg/devicemanager/detector):
    • Extract pure evaluateDeviceUnhealthy health-evaluation helpers in the remaining 7 vendor detector packages (amd, hygon, iluvatar, metax, mthreads, nvidia, thead) and add table-driven unit tests covering healthy, unhealthy, and unreadable/unsupported cases across all 9 manufacturers.
    • Add TestAlignDeviceGroups_RetainsUnhealthyAcceleratorsAcrossAllManufacturers to verify that an 8-card group with 1 unhealthy card retains all 8 cards through alignDeviceGroups with Status.Unhealthy preserved on the faulty card.
  • T3 (docs/modules/devices/preflight.md, docs/README.md, specs/):
    • Document the device-health check row, the <Name> x<Total> (<Unhealthy> unhealthy) detection detail suffix, and in-Pod kubectl exec inspection using gpustack-operator device-manager detect and gpustack-operator device-manager preflight --manufacturer=<manufacturer>.

Which issue(s) this PR links to:

NONE

Special notes for your reviewer:

  • Spec: specs/2026-10-29-preflight-unhealthy-accelerators.md
  • Verified unit tests (go test -race ./pkg/devicemanager/preflight/... ./pkg/devicemanager/detector/...), make lint, make lint docs, and packaged container exec (detect + preflight) runs across AMD (2 × Radeon RX 7800 XT), NVIDIA (2 × GeForce RTX 4090), and Ascend (8 × 910B2).

Does this PR introduce a user-facing change?

Report unhealthy accelerators in `device-manager preflight` via a `device-health` (`unavailable`) check row and `(<N> unhealthy)` detection detail suffix across all manufacturers, and skip capability probes on faulty cards while still preflighting healthy cards on the same node.

…obes

- add capDeviceHealth and annotate detection detail with unhealthy counts
- emit unavailable device-health checks for unhealthy accelerators
- filter unhealthy accelerators out of driver, slice, and management checks
- add unit tests for detection detail, device-health checks, and filtering

Task 1 of preflight-unhealthy-accelerators.

Signed-off-by: thxCode <thxcode0824@gmail.com>
…oss all manufacturers

- extract evaluateDeviceUnhealthy helpers in amd, hygon, iluvatar, metax, mthreads, nvidia, and thead
- add table-driven TestEvaluateDeviceUnhealthy unit tests for each manufacturer
- verify alignDeviceGroups retains unhealthy accelerators across all 9 manufacturers

Task 2 of preflight-unhealthy-accelerators.

Signed-off-by: thxCode <thxcode0824@gmail.com>
Document the device-health capability row, the (<N> unhealthy) detection
detail suffix, and kubectl exec inspection using detect and preflight
inside a running Device Manager Pod.

Task 3 of preflight-unhealthy-accelerators.

Signed-off-by: thxCode <thxcode0824@gmail.com>
@gpustack-code-review

gpustack-code-review Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 1 issue(s) in this PR.

  • ✅ Successfully posted inline: 0 comment(s)
  • 📋 Routed to summary by policy: 1 comment(s)

maintainability · low

📄 pkg/devicemanager/detector/mthreads/device.go (L348-L355)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category maintainability)

Behavioral change in this otherwise mechanical refactor: MonitorAccelerator previously logged a failed GetEccErrorCounter call at V(3) ("failed to get device memory ECC error counter"); the extracted isDeviceUnhealthy helper now swallows the error entirely. Since evaluateDeviceUnhealthy treats a failed query as healthy, transient vendor-library failures on the monitoring path become completely invisible. Consider logging inside isDeviceUnhealthy when !ret.IsSuccess(), or pass a logger through.

Reviewed 7706873..1501c7b only; earlier commits in this PR were reviewed in a previous run.

…rers

Signed-off-by: thxCode <thxcode0824@gmail.com>
@thxCode

thxCode commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

Review Feedback Summary

Source Location Verdict Outcome
IC_kwDOSVaals8AAAABgurt3w (gpustack-code-review summary #1) pkg/devicemanager/detector/mthreads/device.go (L348-L355) False positive (intentional alignment) Not changed. None of the other 8 vendor detectors (amd, ascend, cambricon, hygon, iluvatar, metax, nvidia, thead) nor mthreads.DetectAccelerator log an error when the ECC or health query returns !ret.IsSuccess(). On cards where ECC is disabled or unsupported by the hardware/driver, GetEccErrorCounter returns a non-success status on every polling round; logging it as an error in MonitorAccelerator emits repeated error logs on every monitoring tick for cards operating normally without ECC. isDeviceUnhealthy intentionally aligns mthreads with all other detectors.

@thxCode
thxCode merged commit a4b4638 into main Oct 29, 2026
8 checks passed
@thxCode
thxCode deleted the preflight-unhealthy branch October 29, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant