[server] Recover log and index file for unclean shutdown #1749
+495
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LiebingYu
force-pushed
the
fix-corrupt-index-new
branch
2 times, most recently
from
b9b5f2d to
ceaf704
Compare
|
Ready for CR @wuchong @swuferhong |
swuferhong
reviewed
Sep 25, 2025
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
LiebingYu
force-pushed
the
fix-corrupt-index-new
branch
from
ceaf704 to
1e8a582
Compare
wuchong
reviewed
Nov 4, 2025
| // can potentially skip over more segments. | ||
| LogTablet.rebuildWriterState( | ||
| writerStateManager, logSegments, 0, segment.getBaseOffset(), false); | ||
| int bytesTruncated = segment.recover(); |
There was a problem hiding this comment.
Do we need to pass writerStateManager into segment.recover() to update writer state like how kafka does? cc @swuferhong
| segment.getFileLogRecords().file().getAbsoluteFile()); | ||
| } | ||
| recoverSegment(segment); | ||
| } |
There was a problem hiding this comment.
Should we catch CorruptIndexException and recover segment for this case like how kafka does? And we should rethrow other exceptions in else branch.
There was a problem hiding this comment.
- Kafka throw
CorruptIndexExceptionbecause Kafka will do sanity check for transaction file which may throwCorruptIndexException. But here in fluss we don't do sanity check of index files insegment.sanityCheck. So I think we don't need to catchCorruptIndexExceptionhere. - The caller of this function will take care of the IOException, maybe we don't need to process other IOException here?
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/test/java/org/apache/fluss/server/log/LogLoaderTest.java
Show resolved
Hide resolved
fluss-server/src/main/java/org/apache/fluss/server/log/LogSegment.java
Outdated
Show resolved
Hide resolved
fluss-server/src/main/java/org/apache/fluss/server/log/LogLoader.java
Outdated
Show resolved
Hide resolved
|
I added a commit to fix the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #1716
Brief change log
Tests
API and Format
Documentation