iceberg: fix spec inconsistency in manifest list files_count #24602
+118
−28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andrwng
force-pushed
the
datalake-manifest-list-fix
branch
2 times, most recently
from
b89b3ed to
68397b0
Compare
The schema we are using was pulled some time ago and appears to be outdated. The Apache Iceberg Java implementation has since renamed added_data_files_count and friends to added_files_count, to match the documented spec. This meant that after updating the table with an external non-Redpanda writer, Redpanda wouldn't be able to download the current manifest list when appending and get stuck, complaining about an EOF (presumably the Avro C++ library throws this when there is an unknown field). I suspect that this may have also been the cause of an EOF seen when trying to read a manifest list with BigQuery: Error while reading data, error message: The Apache Avro failed to read data with the following error: EOF reached File: [...]/metadata/snap-[...]-0.avro The old names are added as an alias to ensure Redpanda can still download Iceberg manifest lists from 24.3.
Adds a simple upgrade test to go from 24.3 to HEAD, ensuring progress can be made. A recent change to our Iceberg manifest list schema changed the name of one field, adding the old name as an alias. Without that aliasing, this new test would fail.
andrwng
force-pushed
the
datalake-manifest-list-fix
branch
from
68397b0 to
fe6155c
Compare
andrwng
changed the title
Retry command for Build#59894please wait until all jobs are finished before running the slash command |
CI test resultstest results on build#59894
|
|
Hmm I wonder if we should backport this. Technically this is a breaking change that will prevent downgrades to earlier 24.3 versions. Or is backward compat also working thanks to aliases? |
Yeah good thought. You're right, this will prevent Iceberg from working following a rollback, even with the alias, because 24.3.prev won't be able to read the manifest lists with the new Avro schema. That said, I lean towards it being justifiable -- it's a correctness fix, especially for a beta feature. |
ztlpn
approved these changes
Dec 23, 2024
| @@ -134,6 +134,31 @@ def test_avro_schema(self, cloud_storage_type, query_engine): | |||
| assert spark_describe_out == spark_expected_out, str( | |||
| spark_describe_out) | |||
|
|
|||
| @cluster(num_nodes=4) | |||
| @matrix(cloud_storage_type=[CloudStorageType.S3]) | |||
There was a problem hiding this comment.
nit: will the test fail or be skipped when running on e.g. azure?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The schema we are using was pulled some time ago and appears to be outdated. The Apache Iceberg Java implementation has since renamed added_data_files_count and friends to added_files_count, to match the documented spec.
This meant that after updating the table with an external non-Redpanda writer, Redpanda wouldn't be able to download the current manifest list when appending and get stuck, complaining about an EOF (presumably the Avro C++ library throws this when there is an unknown field).
I suspect that this may have been the cause of an EOF seen when trying to read a manifest list with BigQuery:
The old names are added as an alias to ensure Redpanda can still download Iceberg manifest lists from 24.3. This PR also adds an upgrade test to validate that this alias is actually required, rather than simply renaming the field. Without the alias, the upgrade test fails.
Backports Required
Release Notes
Bug Fixes