Move metadata queries to DuckLakeMetadataManager::Query/Execute#895
Draft
qsliu2017 wants to merge 5 commits into
Draft
Move metadata queries to DuckLakeMetadataManager::Query/Execute#895qsliu2017 wants to merge 5 commits into
qsliu2017 wants to merge 5 commits into
Conversation
qsliu2017
force-pushed
the
move-metadata-queries
branch
from
5e1bd18 to
f0c7a7c
Compare
DuckLakeTransaction::Query now only handles DuckDB-internal queries (ATTACH, duckdb_secrets, read_blob, etc). All metadata queries go through DuckLakeMetadataManager::Query (SELECT) or Execute (mutations). - Add FillSnapshotArgs, FillSnapshotCommitArgs, FillCatalogArgs static helpers to DuckLakeMetadataManager for template replacement - Add Query/Execute overloads: (string) and (DuckLakeSnapshot, string) - Remove DuckLakeTransaction::Query(DuckLakeSnapshot, string) overload - Update ~70 call sites in ducklake_metadata_manager.cpp - Update 3 call sites in ducklake_flush_inlined_data.cpp - Update PostgresMetadataManager to match new signatures Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qsliu2017
force-pushed
the
move-metadata-queries
branch
from
f0c7a7c to
6d01f22
Compare
Resolve conflicts to maintain metadata query routing through DuckLakeMetadataManager::Query/Execute instead of direct DuckLakeTransaction::Query calls. Incorporate upstream additions: logging/timing in Transaction::Query, partitioned flush logic, roaring bitmap deletion vectors, ORDER BY in inlined data reads, and improved error handling. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use GenerateUUID() instead of SQL UUID() for schema initialization, which is not available on all metadata backends (e.g. Postgres). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qsliu2017
force-pushed
the
move-metadata-queries
branch
from
0e5a03b to
e23c673
Compare
Replace DuckDB-specific LIST(STRUCT_PACK(...)) with the virtual ListAggregation method so Postgres gets json_agg(json_build_object(...)). Add LoadInlinedDeletions virtual for parsing the result (DuckDB uses ListValue/StructValue, Postgres uses yyjson). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
qsliu2017
force-pushed
the
move-metadata-queries
branch
4 times, most recently
from
eca5e46 to
ce26e4c
Compare
qsliu2017
force-pushed
the
move-metadata-queries
branch
from
ce26e4c to
0f1d86f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DuckLakeTransactiontoDuckLakeMetadataManager(~70 call sites inducklake_metadata_manager.cpp, 3 inducklake_flush_inlined_data.cpp), soDuckLakeTransaction::Queryis now only for DuckDB-internal queriesFillSnapshotArgs,FillSnapshotCommitArgs,FillCatalogArgs) fromDuckLakeTransactiontoDuckLakeMetadataManageras static helpersQuery/Executesignatures: add(string)and(DuckLakeSnapshot, string)overloads onDuckLakeMetadataManager, removeDuckLakeTransaction::Query(DuckLakeSnapshot, string)PostgresMetadataManagerto override the new no-snapshotQuery/Executesignatures, and add overrides forListAggregation,CastStatsToTarget,CastColumnToTarget,LoadTags,LoadInlinedDataTables,LoadMacroImplementations,TransformInlinedDataPart of the work from #731 (item 2). This makes it easier for custom metadata managers (e.g. pg_ducklake) to override query execution.
🤖 Generated with Claude Code