-
Notifications
You must be signed in to change notification settings - Fork 425
feat: introduce EnterStagingRequest for RegionEngine
#7261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
7c4ec53 to
4400b66
Compare
4400b66 to
7a3f9f6
Compare
c512891 to
bb51c1d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR introduces the EnterStagingRequest API for the RegionEngine, enabling safe partition rule changes through a carefully orchestrated staging mode transition. The implementation adds a new intermediate EnteringStaging state that stalls writes during the transition, flushes memtables with the old partition expression, creates isolated staging manifests with the new partition expression, and then resumes writes in staging mode.
Key changes include:
- New
EnterStagingRequestAPI andRegionLeaderState::EnteringStagingstate for safe partition expression updates - Modified manifest update signature to use
is_staging: boolparameter instead of region state - Enhanced flush handling with new
FlushReason::EnterStagingvariant andis_stagingparameter propagation
Reviewed Changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/store-api/src/region_request.rs |
Adds EnterStagingRequest struct and extends RegionRequest enum with EnterStaging variant |
src/mito2/src/worker/handle_enter_staging.rs |
New file implementing core enter staging logic including request handling, manifest creation, and result processing |
src/mito2/src/worker/handle_write.rs |
Adds write stalling support for EnteringStaging state to queue requests during transition |
src/mito2/src/region.rs |
Adds EnteringStaging state, staging_partition_expr field, and state transition methods |
src/mito2/src/manifest/manager.rs |
Changes update() signature to use is_staging boolean, adds staging manifest tracking and clear_staging_manifests() |
src/mito2/src/manifest/checkpointer.rs |
Simplifies maybe_do_checkpoint() by removing state parameter (staging check moved to caller) |
src/mito2/src/flush.rs |
Adds FlushReason::EnterStaging variant and is_staging field to RegionFlushTask |
src/mito2/src/worker/handle_flush.rs |
Propagates is_staging parameter and updates flush handling to skip metadata updates in staging mode |
src/mito2/src/worker/handle_manifest.rs |
Updates all update_manifest() calls to pass is_staging parameter |
src/mito2/src/worker/handle_alter.rs |
Updates flush task creation to pass is_staging parameter |
src/mito2/src/worker.rs |
Adds EnterStaging request handling and background notification processing |
src/mito2/src/region/opener.rs |
Initializes staging_partition_expr field and refactors manifest options creation |
src/mito2/src/manifest/storage.rs |
Adds manifest_dir() helper function |
src/mito2/src/error.rs |
Adds StagingPartitionExprMismatch error for partition expression validation |
src/mito2/src/engine/staging_test.rs |
Comprehensive tests for enter staging functionality including partition expression validation and write stalling |
src/mito2/src/engine/listener.rs |
Adds NotifyEnterStagingResultListener for test coordination |
src/mito2/src/request.rs |
Adds EnterStagingRequest to worker request types and EnterStagingResult for background notifications |
src/metric-engine/src/engine.rs |
Adds EnterStaging request handling for physical regions |
src/datanode/src/region_server.rs |
Maps EnterStaging request to RegionChange::None |
| Test files (multiple) | Updates existing tests to use new manifest update signature |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
9a60b14 to
c513d6e
Compare
| if let Err(e) = region.exit_entering_staging() { | ||
| error!(e; "Failed to exit entering staging after failed to clear staging manifest files"); | ||
| } | ||
| sender.send(Err(e)); | ||
| return; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to send result to the worker? So it can handle stalled requests.
Signed-off-by: WenyXu <[email protected]>
Signed-off-by: WenyXu <[email protected]>
Signed-off-by: WenyXu <[email protected]>
Signed-off-by: WenyXu <[email protected]>
c513d6e to
4afbb3c
Compare
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
#6558
What's changed and what's your intention?
This PR introduces
EnterStagingRequestas a new region operation that enables safe partition rule changes by transitioning regions into staging mode with isolated manifest management. This allows zero-downtime partition expression updates while maintaining data consistency.The
EnterStagingRequestfollows a carefully designed flow:FlushReason::EnterStagingStagingmodePR Checklist
Please convert it to a draft if some of the following conditions are not met.