[v24.3.x] iceberg: fix spec inconsistency in manifest list files_count #24650
+118
−28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The schema we are using was pulled some time ago and appears to be outdated. The Apache Iceberg Java implementation has since renamed added_data_files_count and friends to added_files_count, to match the documented spec. This meant that after updating the table with an external non-Redpanda writer, Redpanda wouldn't be able to download the current manifest list when appending and get stuck, complaining about an EOF (presumably the Avro C++ library throws this when there is an unknown field). I suspect that this may have also been the cause of an EOF seen when trying to read a manifest list with BigQuery: Error while reading data, error message: The Apache Avro failed to read data with the following error: EOF reached File: [...]/metadata/snap-[...]-0.avro The old names are added as an alias to ensure Redpanda can still download Iceberg manifest lists from 24.3. (cherry picked from commit a3e6880)
Adds a simple upgrade test to go from 24.3 to HEAD, ensuring progress can be made. A recent change to our Iceberg manifest list schema changed the name of one field, adding the old name as an alias. Without that aliasing, this new test would fail. (cherry picked from commit 59cc2c3)
Retry command for Build#60094please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#60094
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport of PR #24602