feat(sandbox): add CheckpointRestore upgrade strategy with UpgradeControl#670
Merged
Merged
Conversation
…trol - Extract upgrade state machine from commonControl into UpgradeControl struct (PreUpgrade → Checkpointing → UpgradePod → PostUpgrade → Succeeded) - Add CheckpointRestore upgrade policy type to Sandbox and SandboxUpdateOps CRDs - Add EnsureCheckpointForUpgrade, GetCheckpointIDForUpgrade, CleanupForUpgrade methods to CheckpointControl for upgrade-specific checkpoint operations - Filter out Checkpoints with DeletionTimestamp in listCheckpointsForSandbox to avoid using stale checkpoints being deleted - Make checkpoint ID annotation key configurable via --checkpoint-id-annotation-key controller flag; remove hardcoded AnnotationCheckpointID constant - Add unit tests for DeletionTimestamp filtering and annotation key behavior Signed-off-by: liheng <liheng.zms@alibaba-inc.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Move CleanupForUpgrade from performRecreateUpgrade (UpgradePod phase) to after PostUpgrade succeeds. This ensures checkpoint CRs are retained for debugging if PostUpgrade fails and needs retry. Signed-off-by: liheng <liheng.zms@alibaba-inc.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #670 +/- ##
==========================================
+ Coverage 80.06% 80.34% +0.27%
==========================================
Files 229 231 +2
Lines 17862 18057 +195
==========================================
+ Hits 14302 14507 +205
+ Misses 2975 2963 -12
- Partials 585 587 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add comprehensive unit tests for upgrade_control.go and checkpoint.go: - TestEnsureCheckpointForUpgrade: 6 cases covering create/wait/succeed/fail/error paths - TestGetCheckpointIDForUpgrade: 4 cases covering with/without ID, no CPs, list error - TestCleanupForUpgrade: 4 cases covering delete all, no-op, list/delete errors - TestEnsureSandboxUpgraded_CheckpointRestore: 5 cases covering full CheckpointRestore lifecycle - TestPerformRecreateUpgrade_ContainerStatuses: 4 cases covering Waiting/Terminated/PodInitializing/ContainerCreating - TestPerformRecreateUpgrade_CheckpointRestore_CreatePod: verifies checkpoint ID is used during pod creation - TestExecuteUpgradeAction_NilAction/NilPod: covers nil action and nil pod defensive paths - TestHasUpgradeAction: 7 cases covering all lifecycle/action combinations Coverage improvement: - checkpoint.go new functions: 0% -> 100% - EnsureSandboxUpgraded: 70.2% -> 95.2% - performRecreateUpgrade: 61.5% -> 88.5% - executeUpgradeAction: 91.7% -> 100% - hasUpgradeAction: 88.9% -> 100% - Overall core package: 87.7% -> 95.2% Signed-off-by: liheng <liheng.zms@alibaba-inc.com>
Signed-off-by: liheng <liheng.zms@alibaba-inc.com>
|
@zmberg: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
furykerry
reviewed
Jul 20, 2026
…immutability validation - Fix CheckpointRestore strategy to enter full upgrade lifecycle (PreUpgrade -> Checkpointing -> UpgradePod -> PostUpgrade) by extending upgrade trigger condition and inplace update guard in sandbox_controller and common_control - Abstract RequiresPodReplacementUpgrade helper to deduplicate Recreate || CheckpointRestore condition check across packages - Propagate checkpoint CR name through createCheckpoint and EnsureCheckpointForUpgrade return values, surface it in upgrade condition message and klog for better observability - Add validating webhook to reject container image changes in patch when UpdateStrategy.Type is CheckpointRestore (checkpoint restore requires image immutability) Signed-off-by: liheng <liheng.zms@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the
CheckpointRestoreupgrade strategy for Sandboxes, enabling zero-downtime upgrades by checkpointing the sandbox writable layer before pod recreation and restoring it afterward.Key Changes
New: UpgradeControl state machine (
upgrade_control.go)commonControlinto a dedicatedUpgradeControlstructPreUpgrade → Checkpointing → UpgradePod → PostUpgrade → SucceededCheckpointingstate usesfallthroughto seamlessly transition toUpgradePodonce checkpoint completesAPI changes
CheckpointRestoreupgrade policy type toSandboxandSandboxUpdateOpsCRDsCheckpointTypePodInfocheckpoint type label for upgrade-specific checkpointsCheckpointControl enhancements (
checkpoint.go)EnsureCheckpointForUpgrade: creates a Checkpoint CR for the sandbox being upgradedGetCheckpointIDForUpgrade: retrieves the checkpoint ID from the latest completed checkpointCleanupForUpgrade: cleans up checkpoint resources after upgrade completeslistCheckpointsForSandbox: filters out Checkpoints with non-nilDeletionTimestampto avoid using stale checkpoints being deletedConfigurable checkpoint annotation key
AnnotationCheckpointIDconstant fromannotations.go--checkpoint-id-annotation-keycontroller flag; when empty, no checkpoint ID annotation is written to podsPodControl.CreatePodguards against empty key/ID before writing annotationSandboxUpdateOps integration (
patch.go)CheckpointRestoreupgrade policyTest Coverage
TestListCheckpointsForSandboxTestCreatePodCheckpointAnnotationwith 4 cases covering annotation key configuration behaviorcommon_control_test.goandupgrade_control_test.goReview Guidance
upgrade_control.gostate machine logic, especially thefallthroughfromCheckpointingtoUpgradePodlistCheckpointsForSandboxDeletionTimestamp filtering is correct