Skip to content

docs(testing): recover interrupted Nebius cluster creation - #744

Merged
thxCode merged 1 commit into
mainfrom
docs/nebius-partial-apply-recovery
Oct 7, 2026
Merged

thxCode merged 1 commit into
mainfrom
docs/nebius-partial-apply-recovery

Conversation

@thxCode

@thxCode thxCode commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind documentation
/area testing

What this PR does / why we need it:

An interrupted apply can leave a Nebius node group present while its Terraform address is absent. A plain destroy can then miss that object.

  • Document how to establish ownership and import the existing group into its planned address before cleanup.
  • Require the historical root and descendant checklist before destroy, followed by exact per-ID absence and preservation of the original inventory.
  • Explain how to inspect drain failures and use a separate cleanup kubeconfig after the destroy provisioner removes the local context. Removing blocking PDBs is limited to an isolated test cluster being removed in full.

Which issue(s) this PR links to:

NONE

Special notes for your reviewer:

  • Documentation only. Terraform and Operator behavior are unchanged.
  • The recovery steps were exercised after a preemptible GPU node group reported NotEnoughResources: the existing group was imported, destroy completed, all 26 recorded owned cloud IDs returned exact NotFound, and five pre-existing IDs survived.
  • GPU inference and Elastic EP resize did not run because the GPU environment never became ready. This PR does not claim physical TP validation.
  • make lint docs and git diff --check passed. The unchanged Operator source previously passed the full race/shuffle/coverage suite in a clean export.
  • Independent read-only review found one ordering issue, corrected before submission. Review of the additional drain recovery paragraph found no further issues.

Does this PR introduce a user-facing change?

NONE

@thxCode thxCode added kind/documentation Improvements or additions to documentation area/testing End-to-end test infrastructure labels Oct 7, 2026
@gpustack-code-review

Copy link
Copy Markdown

✅ OpenCodeReview: Review skipped: no items were selected.

@thxCode

thxCode commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Review feedback check, round 1 of a maximum of 4:

  • Submitted reviews: 0. Inline comments and review threads: 0. Conversation comments: 1. Suppressed findings: 0.
  • The one conversation comment reports that OpenCodeReview selected no items and skipped review. This is an informational outcome; it is not an automated approval of the documentation.
  • Independent read-only review found that the ownership checklist appeared after the destroy command. The steps were reordered before this PR opened. Verification of that correction and the additional drain recovery paragraph found no remaining issues.
  • Docs CI, local make lint docs, make generate, and git diff --check passed. Generation left the tracked tree unchanged.

No review-driven change is needed. The recorded live outcome remains an infrastructure capacity failure before GPU inference, followed by complete cleanup of the owned test resources.

@thxCode
thxCode merged commit 4765e04 into main Oct 7, 2026
2 checks passed
@thxCode
thxCode deleted the docs/nebius-partial-apply-recovery branch October 7, 2026 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing End-to-end test infrastructure kind/documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant