archival: Use read-write fence in the ntp_archiver #24574
+309
−115
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
2e39592 to
a392416
Compare
Retry command for Build#59774please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#59774
test results on build#59783
test results on build#59824
test results on build#59868
test results on build#59928
test results on build#59958
test results on build#59971
test results on build#59988
test results on build#60029
test results on build#60039
test results on build#60048
|
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
a392416 to
ce0cb32
Compare
Retry command for Build#59783please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
ce0cb32 to
06ca1b0
Compare
Retry command for Build#59824please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
06ca1b0 to
63eb5f5
Compare
Retry command for Build#59868please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
63eb5f5 to
b7836dd
Compare
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
b7836dd to
a52b914
Compare
Retry command for Build#59928please wait until all jobs are finished before running the slash command |
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
a52b914 to
9cb55f3
Compare
Retry command for Build#59958please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
Retry command for Build#59988please wait until all jobs are finished before running the slash command |
The snapshot doesn't serialize the applied-offset field of the manifest. This commit fixes this. Signed-off-by: Evgeny Lazin <[email protected]>
The fence is a value which is produced by recording the current offset of the last applied archival STM command. The fence is added to the archival STM config batch. If no commands were applied to the STM the offset of the last applied command will be the same and the configuration batch will be applied. Otherwise, if some command was applied after the record batch was constructed the batch will not be applied. This is essentially a concurrency control mechamism. The commands are already present in the STM but not used as for now. The existing metadata validation mechanism stays.
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
e7116c0 to
83a0abc
Compare
The check is no longer needed because we're using fencing mechanism to prevent concurrency. The check is actually performed when the segment is uploaded. The removed check was used as an optimistic concurrency control mechanism and is now replaced by the RW-fence. Also, deprecate the error code that STM uses to communicate the consistency violation to the caller. The error code is not used anywhere else. Signed-off-by: Evgeny Lazin <[email protected]>
The option was never turned on by default. Our plan was to eventually enable it but it is now replaced with fencing mechanism which is always on and can't be disabled. Signed-off-by: Evgeny Lazin <[email protected]>
This commit enables rw-fence mechanism for housekeeping reuploads. In case if the housekeeping is running on a stale version of the STM snapshot the update will be rejected. Signed-off-by: Evgeny Lazin <[email protected]>
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
83a0abc to
8b33548
Compare
Use rw-fence for both normal retention/GC and spillover. Currently, the upload metadata validation only checks added segments. Because of that there is still some room for concurrent updates during the housekeeping (GC or retention). With rw-fence the metadata updates related to retention or GC will become safer. Signed-off-by: Evgeny Lazin <[email protected]>
Retry command for Build#60029please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
Retry command for Build#60039please wait until all jobs are finished before running the slash command |
Previous implementation of the archival STM had a bug because of which the last applied offset wasn't added to the snapshot. This could potentially make replicas diverge. The solution is to add a feature flag and use the rw-fence mechanism only if all replicas are upgraded. Signed-off-by: Evgeny Lazin <[email protected]>
Lazin
force-pushed
the
pr/use-fencing-mechanism-in-archiver
branch
from
ba45e29 to
9b0a748
Compare
Retry command for Build#60048please wait until all jobs are finished before running the slash command |
|
/ci-repeat 1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The fence is a value which is produced by recording the current offset of the last applied archival STM command. The fence is added to the archival STM config batch. If no commands were applied to the STM the offset of the last applied command will be the same and the configuration batch will be applied. Otherwise, if some command was applied after the record batch was constructed the batch will not be applied.
This is essentially a concurrency control mechanism. The commands are already present in the STM but not used as for now. The existing metadata validation mechanism in the NTP archiver is still working but we no longer need to check the commands in the STM.
Backports Required
Release Notes