Paimon's function about compaction and partial-update #751
Unanswered
yuzhangqin55-bit
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
In my business scenarios,I need to use 8 tasks write to the same table, and I also need only a few columns keep the first non-null value, other be covered.
Now,I set table as:
WITH (
'bucket' = '40',
'path' = 's3://yj-datalake-test/paimon/qinyuzhang.db/Mt5HistoryTradeRecordModel',
'write-mode' = 'WRITE_ONLY',
'merge-engine' = 'partial-update',
'bucket-key' = 'b_id,s_id'
)
and run 8 tasks to write,but it prints logs as:
2025-10-09 05:33:13,865 WARN org.apache.flink.runtime.taskmanager.Task [] - Writer : Mt5HistoryTradeRecordModel -> Global Committer : Mt5HistoryTradeRecordModel -> end: Writer (1/1)#677 (70647f047568adcfd35f0ed8f64fd97d_f6dc7f4d2283f4605b127b9364e21148_0_677) switched from RUNNING to FAILED with failure cause:
java.lang.RuntimeException: File deletion conflicts detected! Give up committing.
Don't panic!
Conflicts during commits are normal and this failure is intended to resolve the conflicts.
Conflicts are mainly caused by the following scenarios:
You'll probably see different base commit user and current commit user below.
You can use https://paimon.apache.org/docs/master/maintenance/dedicated-compaction#dedicated-compaction-job to support multiple writing.
The job will fail continuously in this scenario to protect metadata from corruption.
You can either recover from the latest savepoint, or you can revert the table to the snapshot corresponding to the old savepoint.
Base commit user is: d0d36272-9541-497b-8710-0ad75295d42b; Current commit user is: d0d36272-9541-497b-8710-0ad75295d42b
So table does not save all data.
How should I set up the table to eliminate the problem and retain all the data?
Help me thanks a lot.
Beta Was this translation helpful? Give feedback.
All reactions