Fix NodeStateManager lost update problem #25069
Open
+60
−34
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changing "NodeState" to "Worker State change" clarifies that the logs refer to the worker's state transitions and makes the log consistent with messages in ServerInfoResource. Consistent logs are easier to grep.
The NodeStateManager's drain implementation suffered from a lost update problem. A rapid sequence of drain/active/drain requests could cause the worker to prematurely exit a new draining state, ignoring in-flight tasks, because it failed to distinguish between old and new drain operations. This was resolved by adding versioning to the node state, ensuring each state change request is uniquely identified and processed to completion. Each time the worker state is changed, the version number increases. So even if the state changes quickly from DRAINING,1 to ACTIVE,2 and back to DRAINING,3, the handling code can distinguish which state it was called for. This ensures the handling code only completes a drain action when it's working on the correct (latest) version of the request.
losipiuk
reviewed
Feb 25, 2025
|
|
||
| private record VersionedState(NodeState state, long version) | ||
| { | ||
| private static final AtomicLong versionProvider = new AtomicLong(0); |
There was a problem hiding this comment.
put it in NodeStateManager and make non-static
There was a problem hiding this comment.
Also it does not need to be atomic as it is accessed from synchronized blocks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
The NodeStateManager's drain implementation suffered from a lost update problem. A rapid sequence of drain/active/drain requests could cause the worker to prematurely exit a new draining state, ignoring in-flight tasks, because it failed to distinguish between old and new drain operations.
This was resolved by adding versioning to the node state, ensuring each state change request is uniquely identified and processed to completion. Each time the worker state is changed, the version number increases. So even if the state changes quickly from DRAINING,1 to ACTIVE,2 and back to DRAINING,3, the handling code can distinguish which state it was called for. This ensures the handling code only completes a drain action when it's working on the correct (latest) version of the request.
Additional context and related issues
Example:
The ACTIVE was not observed by all the checks here. The worker will be in a NEW DRAINING state, with new tasks to drain, it should not continue to execute previous draining/drained logic.
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: