Issue 3908 - Send asynchronous compaction commits to new commit lambda from compaction task #4131
Conversation
| @@ -152,6 +152,11 @@ sleeper.table.compaction.job.id.assignment.commit.async=true | |||
| # instance property. | |||
| sleeper.table.compaction.job.commit.async=true | |||
|
|
|||
| # If true, asynchronous commits will be batched if they are enabled. The commits will be gathered into | |||
| # a larger transaction per Sleeper table. If false, the commits will sent directly to the commit | |||
| # lambda. | |||
There was a problem hiding this comment.
This would be clearer as:
"This property affects whether commits of compaction jobs are batched before being sent to the state store commit queue to be applied by the committer lambda. If this property is true and asynchronous commits are enabled then commits of compactions will be batched. If this property is false and asynchronous commits are enabled then commits of compactions will not be batched and will be sent directly to the committer lambda."
| @@ -1462,6 +1462,10 @@ sleeper.default.compaction.job.id.assignment.commit.async=true | |||
| # finished jobs directly to the state store. | |||
| sleeper.default.compaction.job.commit.async=true | |||
|
|
|||
| # This is the default for whether the asynchronous compaction commits will be sent to the batcher if | |||
| # enabled. | |||
There was a problem hiding this comment.
This would be clearer as:
"This property is the default for whether commits of compaction jobs are batched before being sent to the state store commit queue to be applied by the committer lambda. If this property is true and asynchronous commits are enabled then commits of compactions will be batched. If this property is false and asynchronous commits are enabled then commits of compactions will not be batched and will be sent directly to the committer lambda. This property can be overridden for individual tables."
| @@ -294,6 +294,12 @@ public interface TableDefaultProperty { | |||
| .defaultValue("true") | |||
| .validationPredicate(SleeperPropertyValueUtils::isTrueOrFalse) | |||
| .propertyGroup(InstancePropertyGroup.TABLE_PROPERTY_DEFAULT).build(); | |||
| UserDefinedInstanceProperty DEFAULT_COMPACTION_JOB_ASYNC_BATCHING = Index.propertyBuilder("sleeper.default.compaction.job.async.commit.batching") | |||
| .description("This is the default for whether the asynchronous compaction commits will be sent " + | |||
| "to the batcher if enabled.") | |||
| .defaultProperty(DEFAULT_COMPACTION_JOB_ASYNC_BATCHING) | ||
| .description("If true, asynchronous commits will be batched if they are enabled. The commits will be " + | ||
| "gathered into a larger transaction per Sleeper table. If false, the commits will sent directly " + | ||
| "to the commit lambda. ") |
|
Approved pending above comments. |
Make sure you have checked all steps below.
Issue
PR"
Tests
Documentation
separate issue for that below.