Skip to content

[v24.3.x] iceberg: fix spec inconsistency in manifest list files_count#24650

Merged
andrwng merged 2 commits intoredpanda-data:v24.3.xfrom
vbotbuildovich:backport-pr-24602-v24.3.x-82
Jan 8, 2025
Merged

[v24.3.x] iceberg: fix spec inconsistency in manifest list files_count#24650
andrwng merged 2 commits intoredpanda-data:v24.3.xfrom
vbotbuildovich:backport-pr-24602-v24.3.x-82

Conversation

@vbotbuildovich
Copy link
Copy Markdown
Collaborator

@vbotbuildovich vbotbuildovich commented Dec 23, 2024 •
edited by andrwng
Loading

Backport of PR #24602

  • Added 7ec7c6d
  • Removed the upgrade test since 24.3 is the first version with Iceberg integration

@vbotbuildovich vbotbuildovich added this to the v24.3.x-next milestone Dec 23, 2024
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label Dec 23, 2024
@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

vbotbuildovich commented Dec 24, 2024 •
edited
Loading

Retry command for Build#60094

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/datalake/datalake_e2e_test.py::DatalakeE2ETests.test_upload_after_external_update@{"cloud_storage_type":1}

@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

vbotbuildovich commented Dec 24, 2024 •
edited
Loading

CI test results

test results on build#60094
test_id test_kind job_url test_status passed
gtest_raft_rpunit.gtest_raft_rpunit unit https://buildkite.com/redpanda/redpanda/builds/60094#0193f598-ae63-4c65-8cb0-41cc29ff82dc FLAKY 1/2
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60094#0193f5de-b7e9-4e27-a0e9-fb13bbb12a81 FAIL 0/1
rptest.tests.datalake.datalake_e2e_test.DatalakeE2ETests.test_upload_after_external_update.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60094#0193f5f1-6ec6-4417-93a7-6d4b32580e2b FAIL 0/1
test results on build#60133
test_id test_kind job_url test_status passed
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a19-49d2-8d6c-3202b4822d74 FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=False.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a18-4539-9613-5fae27a32dda FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=False.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a19-49d2-8d6c-3202b4822d74 FAIL 0/1
rptest.tests.random_node_operations_test.RandomNodeOperationsTest.test_node_operations.enable_failures=True.mixed_versions=True.with_tiered_storage=True.with_iceberg=False.cloud_storage_type=CloudStorageType.S3 ducktape https://buildkite.com/redpanda/redpanda/builds/60133#0193fb30-0a18-4539-9613-5fae27a32dda FAIL 0/6
test results on build#60374
test_id test_kind job_url test_status passed
rptest.tests.archive_retention_test.CloudArchiveRetentionTest.test_delete.cloud_storage_type=CloudStorageType.ABS.retention_type=retention.ms ducktape https://buildkite.com/redpanda/redpanda/builds/60374#01944314-e9fd-4ad7-ac0f-b55db607d189 FAIL 0/1

@andrwng andrwng force-pushed the backport-pr-24602-v24.3.x-82 branch from 038e933 to 2cfb307 Compare December 24, 2024 18:52
@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

vbotbuildovich commented Dec 24, 2024

Retry command for Build#60129

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}

@andrwng andrwng force-pushed the backport-pr-24602-v24.3.x-82 branch from 2cfb307 to d9c34cb Compare December 24, 2024 22:35
@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

vbotbuildovich commented Dec 25, 2024 •
edited
Loading

Retry command for Build#60133

please wait until all jobs are finished before running the slash command


/ci-repeat 1
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":false}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":false,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}
tests/rptest/tests/random_node_operations_test.py::RandomNodeOperationsTest.test_node_operations@{"cloud_storage_type":1,"enable_failures":true,"mixed_versions":true,"with_iceberg":false,"with_tiered_storage":true}

@andrwng
Copy link
Copy Markdown
Contributor

andrwng commented Jan 7, 2025

/ci-repeat

mmaslankaprv and others added 2 commits January 7, 2025 13:39
@mmaslankaprv @andrwng
tests: added escaping table names in queries
3c0903a 
Some characters that may be used in topic names must be treated specialy
in SQL. Table names must be quoted when they contains the special
character. Quotation mark is specific to query engine. Added quotation
for table names for query engines used in tests.

Signed-off-by: Michał Maślanka <michal@redpanda.com>
(cherry picked from commit 8fe08cd)
@andrwng
iceberg: fix inconsistency in manifest list files_count
c5ff7e4 
The schema we are using was pulled some time ago and appears to be
outdated. The Apache Iceberg Java implementation has since renamed
added_data_files_count and friends to added_files_count, to match the
documented spec.

This meant that after updating the table with an external non-Redpanda
writer, Redpanda wouldn't be able to download the current manifest list
when appending and get stuck, complaining about an EOF (presumably the
Avro C++ library throws this when there is an unknown field).

I suspect that this may have also been the cause of an EOF seen when
trying to read a manifest list with BigQuery:

Error while reading data, error message: The Apache Avro failed to read data with the following error: EOF reached File: [...]/metadata/snap-[...]-0.avro

The old names are added as an alias to ensure Redpanda can still
download Iceberg manifest lists from 24.3.

(cherry picked from commit a3e6880)
@andrwng andrwng force-pushed the backport-pr-24602-v24.3.x-82 branch from d9c34cb to c5ff7e4 Compare January 7, 2025 21:39
@vbotbuildovich
Copy link
Copy Markdown
Collaborator Author

vbotbuildovich commented Jan 8, 2025

Retry command for Build#60374

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/archive_retention_test.py::CloudArchiveRetentionTest.test_delete@{"cloud_storage_type":2,"retention_type":"retention.ms"}

@andrwng andrwng requested a review from ztlpn January 8, 2025 17:18
@andrwng andrwng enabled auto-merge January 8, 2025 17:22
@andrwng andrwng requested a review from bharathv January 8, 2025 17:22
@andrwng andrwng merged commit 7e7683f into redpanda-data:v24.3.x Jan 8, 2025
@piyushredpanda piyushredpanda modified the milestones: v24.3.x-next, v24.3.3 Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrwng andrwng Awaiting requested review from andrwng

@bharathv bharathv Awaiting requested review from bharathv

1 more reviewer

@ztlpn ztlpn ztlpn approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area/redpanda kind/backport PRs targeting a stable branch

Projects

None yet

Milestone

v24.3.3

Development

Successfully merging this pull request may close these issues.

5 participants

@vbotbuildovich @andrwng @ztlpn @piyushredpanda @mmaslankaprv