Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make commit after every batch of inserts optional, with the default not to commit #81

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

forsberg
Copy link
Contributor

For at least once semantics, it is not necessary to commit after every call to put(), as Kafka Connect will commit the kafka Offsets periodically. Doing it periodically will lead to less writes to the __consumer_offsets topic, and hence, less load on the Kafka brokers.

The default is false, i.e not to commit offset after every set of INSERT operations. This is a change in behaviour of Kafka Connect ScyllaDB, but it leads to a significant decrease in number of produce requests on the Kafka brokers, which in production environments with high load is quite visible in Broker CPU consumption metrics.

…ot to commit.

For at least once semantics, it is not necessary to commit after every call to put(), as Kafka
Connect will commit the kafka Offsets periodically. Doing it periodically will lead to less
writes to the __consumer_offsets topic, and hence, less load on the Kafka brokers.

The default is false, i.e not to commit offset after every set of INSERT operations. This
is a change in behaviour of Kafka Connect ScyllaDB, but it leads to a significant decrease
in number of produce requests on the Kafka brokers, which in production environments
with high load is quite visible in Broker CPU consumption metrics.
@forsberg forsberg force-pushed the feature/optional-additional-commit branch from 263686f to f29a0c3 Compare November 15, 2022 12:09
@mykaul mykaul added the enhancement New feature or request label Mar 20, 2023
@mykaul mykaul requested a review from Bouncheck March 20, 2023 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants