[v24.2.x] Fix stepping down on timeout #24708
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `raft::reply_result::follower_busy` is indicating that the follower was unable to process the heartbeat fast enough to generate a response. Renaming the reply from `timeout` will make it less confusing for the reader and differentiate the error code from an RPC timeout. Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 6a1e34b)
Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 95a29db)
Wired raft RPC service handler into Raft fixture to make the tests more accurate and cover the service code with tests. Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 5f69d9b)
Propagating timeout to the node sending RPC request is crucial for accurate testing of Raft implementation. Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 7d33bb5)
mmaslankaprv
force-pushed
the
manual-backport-24590-v24.2.x-643
branch
from
6a30603 to
3259d02
Compare
Retry command for Build#60349please wait until all jobs are finished before running the slash command |
mmaslankaprv
force-pushed
the
manual-backport-24590-v24.2.x-643
branch
from
3259d02 to
60cf6da
Compare
Added a wrapper around the `storage::log` allowing us to inject storage layer failures in Raft fixture tests. Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit f04995a)
When follower is busy it may fail fast processing full heartbeat requests sent by the leader. In this case a follower RPC handler sets the `follower_busy` result in heartbeat_reply. Leader should still treat a follower replica as online in this case. The replica hosting node must be online to reply with the `follower_busy` error. This way we prevent to eager leader step downs when follower replicas are slow. Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 8b57b42)
Signed-off-by: Michał Maślanka <[email protected]> (cherry picked from commit 67e7c6e)
mmaslankaprv
force-pushed
the
manual-backport-24590-v24.2.x-643
branch
from
60cf6da to
683d30a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #24590
Fixes: #24668