[core] Support chain tbl on batch mode #6394
Conversation
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
4 times, most recently
from
7efbc2f to
817cf2a
Compare
| /** Utils for table. */ | ||
| public class ChainTableUtils { | ||
|
|
||
| public static void checkChainTableOptions( |
There was a problem hiding this comment.
Add this in SchemaValidation?
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
817cf2a to
84ee5b3
Compare
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/schema/SchemaValidation.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/io/KeyValueFileReaderFactory.java
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/MergeTreeCompactManager.java
Outdated
Show resolved
Hide resolved
|
Looks good to me +1. Please take another look cc @JingsongLi |
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
5 times, most recently
from
e9c9096 to
90ac914
Compare
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
2 times, most recently
from
b9000a9 to
49c13e0
Compare
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Please add documentation.
| return result.substring(0, Math.min(result.length(), maxLength)); | ||
| } | ||
|
|
||
| public List<String> generateOrderPartValues(InternalRow in) { |
There was a problem hiding this comment.
Why add this? What is difference? You can add a test for this.
There was a problem hiding this comment.
It is hoped to obtain the partition values in the order of the partition fields, and then calculate the dependent partitions according to the unit step size. For example, taking the hourly level as an example, the unit step size of the partition is hours. If starting from 0:00 on 20250811 and ending at 0:00 on 20250812, 24 incremental partitions in hours can be obtained, for the specific usage, please refer to org.apache.paimon.partition.InternalRowPartitionUtils#getDeltaPartitions. Unit tests have been added, please refer to org.apache.paimon.utils.InternalRowPartitionComputerTest#testGenerateOrderPartValues
| return newPathFromName(newFileName(prefix)); | ||
| } | ||
|
|
||
| public Path newPath(String prefix, Path relativeBucketPath) { |
There was a problem hiding this comment.
For chain table, one logical partition may correspond to multiple physical partitions. Therefore, when creating a new path, on the basis of the original api about newPath, both partitions and buckets need to be specified simultaneously. The current interface is not in use for now. It is only for compatibility with the function of creating new path
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
4 times, most recently
from
bc0d580 to
ae17ace
Compare
done |
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
ae17ace to
225f73b
Compare
Co-authored-by: zhoufa <[email protected]>
Stefanietry
force-pushed
the
support_chain_table_on_batch
branch
from
225f73b to
7211870
Compare
3 participants
Purpose
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table
Linked issue
#6313
Tests
see: org.apache.paimon.spark.SparkCatalogWithHiveTest#testChainTable
API and Format
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table
Documentation
Related to PIP: https://cwiki.apache.org/confluence/display/PAIMON/PIP-37%3A+Introduce+Chain+Table