Support retrieving clustering information of Delta Lake tables. #27052
Conversation
Reviewer's GuideThis PR adds support for retrieving clustering information from Delta Lake tables behind a new feature flag. It introduces a session/catalog property to toggle visibility, extends the metadata layer and table handles to carry an optional List of clustered columns, implements logic to parse clustering info from the transaction log (via a new utility and an Operation enum), and updates tests and documentation accordingly. Sequence diagram for retrieving clustered columns from Delta Lake tablesequenceDiagram
participant Session
participant DeltaLakeMetadata
participant TransactionLogAccess
participant TableSnapshot
participant ClusteringMetadataUtil
Session->>DeltaLakeMetadata: getTableHandle(session, ...)
DeltaLakeMetadata->>TransactionLogAccess: getClusteredColumns(fileSystem, tableSnapshot)
TransactionLogAccess->>TableSnapshot: getCachedClusteredColumns()
alt Not cached
TransactionLogAccess->>ClusteringMetadataUtil: getLatestClusteredColumns(fileSystem, tableSnapshot)
ClusteringMetadataUtil-->>TransactionLogAccess: clusteredColumns
TransactionLogAccess->>TableSnapshot: setCachedClusteredColumns(clusteredColumns)
end
TransactionLogAccess-->>DeltaLakeMetadata: clusteredColumns
DeltaLakeMetadata-->>Session: LocatedTableHandle(clusteredColumns)
ER diagram for Delta Lake table properties with clustering infoerDiagram
DELTA_LAKE_TABLE_PROPERTIES {
string location
list partitioned_by
list clustered_by
long checkpoint_interval
string change_data_feed_enabled
string column_mapping_mode
}
DELTA_LAKE_TABLE_HANDLE {
string location
object metadata_entry
object protocol_entry
list clustered_columns
}
DELTA_LAKE_TABLE_PROPERTIES ||--|| DELTA_LAKE_TABLE_HANDLE : "table handle for properties"
DELTA_LAKE_TABLE_HANDLE {
list clustered_columns
}
Class diagram for Delta Lake clustering metadata supportclassDiagram
class DeltaLakeConfig {
- boolean showClusteredColumns
+ boolean isShowClusteredColumns()
+ DeltaLakeConfig setShowClusteredColumns(boolean)
}
class DeltaLakeSessionProperties {
+ static boolean ifShowClusteredColumns(ConnectorSession)
}
class DeltaLakeTableProperties {
+ static final String CLUSTER_BY_PROPERTY
+ static List<String> getClusteredBy(Map<String, Object>)
}
class DeltaLakeTableHandle {
- Optional<List<String>> clusteredColumns
+ Optional<List<String>> getClusteredColumns()
}
class TableSnapshot {
- Optional<List<String>> cachedClusteredColumns
+ Optional<List<String>> getCachedClusteredColumns()
+ void setCachedClusteredColumns(Optional<List<String>>)
}
class TransactionLogAccess {
+ Optional<List<String>> getClusteredColumns(TrinoFileSystem, TableSnapshot)
}
class ClusteringMetadataUtil {
+ static Optional<List<String>> getLatestClusteredColumns(TrinoFileSystem, TableSnapshot)
}
class Operation {
<<enum>>
+ static Operation fromString(String)
}
DeltaLakeConfig --> DeltaLakeSessionProperties
DeltaLakeSessionProperties --> DeltaLakeTableProperties
DeltaLakeTableProperties --> DeltaLakeTableHandle
DeltaLakeTableHandle --> TableSnapshot
TableSnapshot --> TransactionLogAccess
TransactionLogAccess --> ClusteringMetadataUtil
ClusteringMetadataUtil --> Operation
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey there - I've reviewed your changes and they look great!
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location> `plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/clustering/ClusteringMetadataUtil.java:94` </location>
<code_context>
+ REPLACE_TABLE_KEYWORD, CLUSTERING_PARAMETER_KEY,
+ CLUSTER_BY, NEW_CLUSTERING_PARAMETER_KEY);
+
+ private static final ThreadLocal<Map<String, String>> OLD_TO_NEW_RENAMED_COLUMNS = ThreadLocal.withInitial(HashMap::new);
+
+ private ClusteringMetadataUtil()
</code_context>
<issue_to_address>
**issue (bug_risk):** ThreadLocal usage for OLD_TO_NEW_RENAMED_COLUMNS may leak memory if not cleared in all code paths.
If an exception occurs before ThreadLocal removal, it may not be cleared. Use a try-finally block to guarantee cleanup and prevent memory leaks.
</issue_to_address>
### Comment 2
<location> `plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/clustering/ClusteringMetadataUtil.java:249` </location>
<code_context>
+ }
+
+ @VisibleForTesting
+ static void recordRenamedColumns(CommitInfoEntry commitInfoEntry)
+ {
+ String oldName = commitInfoEntry.operationParameters().get(RENAMED_OLD_COLUMN_KEY);
</code_context>
<issue_to_address>
**suggestion:** The logic for updating OLD_TO_NEW_RENAMED_COLUMNS may not handle multiple renames correctly.
The current approach may lose information if a column is renamed multiple times. Please consider tracking all previous names to ensure the mapping remains accurate.
</issue_to_address>
### Comment 3
<location> `plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/clustering/Operation.java:96` </location>
<code_context>
+ CREATE_TABLE_KEYWORD.getOperationName().toLowerCase(), CREATE_TABLE_KEYWORD,
+ REPLACE_TABLE_KEYWORD.getOperationName().toLowerCase(), REPLACE_TABLE_KEYWORD);
+
+ public static Operation fromString(String operationName)
+ {
+ Operation operation = LOWERCASE_NAME_TO_OPERATION.get(operationName.toLowerCase());
</code_context>
<issue_to_address>
**suggestion:** fromString may return UNKNOW_OPERATION for valid but differently-cased or formatted operation names.
Currently, only exact matches are supported, so inputs with extra whitespace or formatting may not be recognized. Consider trimming whitespace or using regex to improve matching robustness.
</issue_to_address>
### Comment 4
<location> `plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeBasic.java:2735-2736` </location>
<code_context>
assertQuery("SELECT * FROM " + sourceTable, sourceTableValues);
}
+ @Test
+ void testShowCreateTableWithClusteredInfo()
+ {
+ Session session = Session.builder(getSession())
</code_context>
<issue_to_address>
**suggestion (testing):** Test for clustered columns in SHOW CREATE TABLE covers both enabled and disabled states.
Please also add a test case for the default configuration (without setting the session property) to confirm expected default behavior.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| REPLACE_TABLE_KEYWORD, CLUSTERING_PARAMETER_KEY, | ||
| CLUSTER_BY, NEW_CLUSTERING_PARAMETER_KEY); | ||
|
|
||
| private static final ThreadLocal<Map<String, String>> OLD_TO_NEW_RENAMED_COLUMNS = ThreadLocal.withInitial(HashMap::new); |
There was a problem hiding this comment.
issue (bug_risk): ThreadLocal usage for OLD_TO_NEW_RENAMED_COLUMNS may leak memory if not cleared in all code paths.
If an exception occurs before ThreadLocal removal, it may not be cleared. Use a try-finally block to guarantee cleanup and prevent memory leaks.
| } | ||
|
|
||
| @VisibleForTesting | ||
| static void recordRenamedColumns(CommitInfoEntry commitInfoEntry) |
There was a problem hiding this comment.
suggestion: The logic for updating OLD_TO_NEW_RENAMED_COLUMNS may not handle multiple renames correctly.
The current approach may lose information if a column is renamed multiple times. Please consider tracking all previous names to ensure the mapping remains accurate.
| CREATE_TABLE_KEYWORD.getOperationName().toLowerCase(), CREATE_TABLE_KEYWORD, | ||
| REPLACE_TABLE_KEYWORD.getOperationName().toLowerCase(), REPLACE_TABLE_KEYWORD); | ||
|
|
||
| public static Operation fromString(String operationName) |
There was a problem hiding this comment.
suggestion: fromString may return UNKNOW_OPERATION for valid but differently-cased or formatted operation names.
Currently, only exact matches are supported, so inputs with extra whitespace or formatting may not be recognized. Consider trimming whitespace or using regex to improve matching robustness.
yangshangqing95
force-pushed
the
retriev-deltalake-clustering-info
branch
2 times, most recently
from
ef44562 to
9163402
Compare
yangshangqing95
force-pushed
the
retriev-deltalake-clustering-info
branch
from
9163402 to
660241a
Compare
|
Is this PR preparatory work for future performance improvements, with no immediate benefit? |
Hi @ebyhr |
|
Could you please share the final solution (= how to improve performance eventually)? |
|
Sure @ebyhr
In simple terms, compared with partitions, Liquid Clustering provides:
Compared with Parquet column statistics, which only store per-file min/max values without any global organization, Liquid Clustering provides a higher-level layout strategy. It ensures that values of clustering keys are physically localized across files, making Parquet’s per-file statistics far more effective for data skipping and reducing the number of files scanned. Once Trino integrates with this metadata, it will be able to perform more accurate file pruning and data skipping, enabling more flexible data organization and clustering, thereby significantly improving the performance of queries filtering on clustering keys. |
|
Thanks for explaining the details. I already know about liquid clustering. I wanted to know the actual follow-up plan (especially read part) for the Delta Lake connector.
The connector already uses stats from the transaction logs. Are you planning to read different metadata in the future? If so, could you elaborate on that? |
|
@yangshangqing95 Would you mind sharing the plan for supporting the write path of liquid clustering? I suspect the read path won’t change much, since Trino already supports pruning with statistics, so reading clustering information doesn’t seem to provide additional benefit in my opinion |
|
Haha, please ignore my long-winded message above.
Coming back to this PR itself — It’s mostly about test data and test code — I hope to cover enough cases. My idea is to break down the larger system into smaller modules or features, which makes it easier to review and identify issues during testing. Open to any discussions or suggestions. |
3 participants
Description
Support retrieving clustering information from Delta Lake tables, controlled by session and configuration settings.
By default, retrieving clustering information is disabled (false).
This serves as a foundation for future integration with the Delta Lake Liquid Clustering feature.
The work to fully support Delta Lake’s Liquid Clustering capability is already planned and in progress.
Additional context and related issues
About Delta Lake Liquid Clustering: https://delta.io/blog/liquid-clustering/
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:
Summary by Sourcery
Enable optional retrieval of clustering information for Delta Lake tables and expose it as a new table property to support future Liquid Clustering integration.
New Features:
Enhancements:
Documentation:
Tests: