[core] Support chain tbl on batch mode #6394
Conversation
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
4 times, most recently
from
7efbc2f to
817cf2a
Compare
| /** Utils for table. */ | ||
| public class ChainTableUtils { | ||
|
|
||
| public static void checkChainTableOptions( |
There was a problem hiding this comment.
Add this in SchemaValidation?
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
817cf2a to
84ee5b3
Compare
| public static void validateChainTableOptions( | ||
| Map<String, String> tableOptions, @Nullable String primaryKeys, boolean partitionTbl) { | ||
| if (Boolean.parseBoolean(tableOptions.get(CoreOptions.CHAIN_TABLE_ENABLED.key()))) { | ||
| Options options = Options.fromMap(tableOptions); |
There was a problem hiding this comment.
I think we should also check that the changelog-producer != ("lookup", "full-compaction")
|
|
||
| public static void validateChainTableOptions( | ||
| Map<String, String> tableOptions, @Nullable String primaryKeys, boolean partitionTbl) { | ||
| if (Boolean.parseBoolean(tableOptions.get(CoreOptions.CHAIN_TABLE_ENABLED.key()))) { |
There was a problem hiding this comment.
change to options.get(CoreOptions.CHAIN_TABLE_ENABLED)
| .withDescription("Specify chain table enable."); | ||
|
|
||
| public static final ConfigOption<String> SCAN_FALLBACK_SNAPSHOT_BRANCH = | ||
| key("scan.fallback-snapshot-branch") |
There was a problem hiding this comment.
if it doesn't have the default value, we have to check this when the user enable the chain table
There was a problem hiding this comment.
This config depends on the actual created branch and operates in sequence with the table creation, so it is designed to be aligned with scan.fallback-branch
| this.dvFactory = dvFactory; | ||
| } | ||
|
|
||
| public KeyValueFileReaderFactory(KeyValueFileReaderFactory factory) { |
There was a problem hiding this comment.
can we introduce a copy method to replace this construct?
There was a problem hiding this comment.
This is used for inheritance scenarios for the member variable initialization, and the member is declared as final
|
|
||
| @Nullable private final RecordLevelExpire recordLevelExpire; | ||
|
|
||
| private final boolean isChainTbl; |
|
Looks good to me +1. Please take another look cc @JingsongLi |
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
5 times, most recently
from
e9c9096 to
90ac914
Compare
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
2 times, most recently
from
b9000a9 to
49c13e0
Compare
| return result.substring(0, Math.min(result.length(), maxLength)); | ||
| } | ||
|
|
||
| public List<String> generateOrderPartValues(InternalRow in) { |
There was a problem hiding this comment.
Why add this? What is difference? You can add a test for this.
There was a problem hiding this comment.
It is hoped to obtain the partition values in the order of the partition fields, and then calculate the dependent partitions according to the unit step size. For example, taking the hourly level as an example, the unit step size of the partition is hours. If starting from 0:00 on 20250811 and ending at 0:00 on 20250812, 24 incremental partitions in hours can be obtained, for the specific usage, please refer to org.apache.paimon.partition.InternalRowPartitionUtils#getDeltaPartitions. Unit tests have been added, please refer to org.apache.paimon.utils.InternalRowPartitionComputerTest#testGenerateOrderPartValues
| return newPathFromName(newFileName(prefix)); | ||
| } | ||
|
|
||
| public Path newPath(String prefix, Path relativeBucketPath) { |
There was a problem hiding this comment.
For chain table, one logical partition may correspond to multiple physical partitions. Therefore, when creating a new path, on the basis of the original api about newPath, both partitions and buckets need to be specified simultaneously. The current interface is not in use for now. It is only for compatibility with the function of creating new path
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
49c13e0 to
91e9697
Compare
Co-authored-by: zhoufa <[email protected]>
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
91e9697 to
3aa1526
Compare
3 participants
Purpose
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table
Linked issue
#6313
Tests
see: org.apache.paimon.spark.SparkCatalogWithHiveTest#testChainTable
API and Format
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table
Documentation
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table