Release v2.2.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-lt/images/sha256-40c1fca45b6c7c66417e9b4010776b6092a8f2ba2cd5bb59e461c4de17b9c384?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.2.1-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.1-kafka/images/sha256-78571eec2d5d9103bf75a993851e051d739cf110b07160e9f294d5d3baf7f494?context=explore

What's Changed

• Update README.md by @subkanthi in #687
• Added log4j2.xml by @subkanthi in #688
• Update logging.md by @subkanthi in #690
• Update logging.md by @subkanthi in #691
• Update quickstart_kafka.md by @subkanthi in #696
• Create incremental_snapshot.md- Document steps to add newer tables. by @subkanthi in #697
• Update README.md to include incremental snapshot document by @subkanthi in #698
• Update incremental_snapshot.md - fix typo by @subkanthi in #699
• Added config with keepermap storage and removed FINAL in CREATE VIEW … by @subkanthi in #695
• Added ClickHouse trademark by @hodgesrm in #702
• Update incremental_snapshot.md by @subkanthi in #706
• Add rrmt check kafka by @Selfeer in #707
• Update incremental_snapshot.md to include schema changes by @subkanthi in #709
• Snapshot out of memory documentation by @subkanthi in #710
• Update production_setup.md for incremental snapshot size by @subkanthi in #711
• 713 numeric datatype in postgres mapped to decimal102 in clickhouse by @subkanthi in #715
• Ignore retrieval of scale/precision for enum data types by @subkanthi in #693
• Added logic to support ALTER TABLE with ALGORITHM by @subkanthi in #719

Full Changelog: 2.2.0...2.2.1

Release v2.2.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.2.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.2.0-lt/images/sha256-c2e2cf6d6dc59043a0cbc097ac908381779a2eaf3508ff2a0062f3beb8f2c4cb?context=explore

Breaking Changes

The environment variable LOGGING_LEVEL is no longer used for setting logging level of the application,from this release it will be set in the Root level="info" line in log4j2.xml.

    <Root level="info" additivity="false">
            <AppenderRef ref="console" />

Examples(Debug)

    <Root level="debug" additivity="false">
            <AppenderRef ref="console" />

Examples(Error)

    <Root level="error" additivity="false">
            <AppenderRef ref="console" />

Logging is configured using https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/log4j2.xml and is mounted to the docker container in https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/docker/clickhouse-sink-connector-lt-service.yml. Refer to https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/logging.md on setup.

What's Changed

• Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #587
• Added log.error statements for all RunTimeException by @subkanthi in #588
• Update development.md by @subkanthi in #604
• Add TesTflows tests support for ReplicatedReplacingMergeTree engine by @Selfeer in #610
• Revert back debezium version to 2.5.0.Beta1 by @subkanthi in #616
• Added support to use user provided rmt delete column by @subkanthi in #593
• Update quickstart README by @hoanganht1k27 in #618
• Fix runners 24 4 by @Selfeer in #621
• Fixed renaming of is_deleted column when the source columns have backticks by @subkanthi in #620
• Added logic to include database name in the logs by @subkanthi in #613
• Added Integration test to validate mapping of MySQL JSON data types t… by @subkanthi in #592
• Fix fails testflows by @Selfeer in #629
• Fix broken CI/CD for TestFlows by @Selfeer in #633
• 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #632
• Resolve 2 2 0 conflicts by @Selfeer in #651
• 631 add database name or UUID to rrmt engine definition when tables are auto created by @subkanthi in #643
• 2.1.0 by @subkanthi in #647
• Database name overrides by @subkanthi in #624
• Remove clickhouse server database by @subkanthi in #614
• Added Integration test to validate mismatch of mysql columns by @subkanthi in #627
• Log exception cause when there is an exception starting connector by @subkanthi in #652
• Added production setup architecture by @subkanthi in #638
• 575 add integration test multiple schema for postgresql by @subkanthi in #579
• Return error code when there is an error retrieving show_replica_status by @subkanthi in #658
• Added logic to OVERRIDE CREATE DATABASE SQL by @subkanthi in #653
• Changes to fix grafana datasource by @subkanthi in #639
• Change MySQL docker image used in Java Integration test from latest t… by @subkanthi in #670
• Added logic to move the CREATE VIEW operation to after the connector … by @subkanthi in #668
• Avoid printing error logs when there are no records in the queue by @subkanthi in #674
• Updated documentation on handling datetime/timestamp fields by @subkanthi in #662
• Upgraded to debezium 2.7.0 and update new function definition for MyS… by @subkanthi in #590

New Contributors

• @hoanganht1k27 made their first contribution in #618

Full Changelog: 2.1.0...2.2.0

Release v2.1.0

Lightweight
Docker: altinity/clickhouse-sink-connector:2.1.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-lt/images/sha256-eb436979852212c7a0f624c1cc2f870ba4b173027be8e21654cdc9179b6d5590?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.1.0-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.1.0-kafka/images/sha256-65d4cbdcdfd159d6d7a1702635cd44da3135fa37dbab637a7a958e3339122b92?context=explore

What's Changed

Breaking Changes.

The configuration clickhouse.server.database is now deprecated with the multiple database support.
By default the source MySQL/postgres database name will be used as the ClickHouse destination database name.

Changes

• Release 2.0.2 by @subkanthi in #510
• Added release notes for 2.0.2 by @subkanthi in #527
• add 1 second delay after query execution by @Selfeer in #537
• Update README.md by @subkanthi in #539
• Update Monitoring.md by @subkanthi in #541
• Change index_granularity to 8192 instead of 8198. by @subkanthi in #534
• Refactor TestFlows tests related to Lightweight by @Selfeer in #543
• Update config.yml to include database.server.id by @subkanthi in #544
• Update Troubleshooting.md by @subkanthi in #545
• Update Monitoring.md to include insert duration query and part log query by @subkanthi in #546
• Removed references to deduplication.policy in kafka configuration by @subkanthi in #547
• Use sequence number + timestamp in non-gtid mode for version column. by @subkanthi in #550
• Added logic to support multiple databases by @subkanthi in #535
• 523 handle scenario when records could be inserted with the same timestampnon gtid mode by @subkanthi in #552
• [528] Added logic to create view for replica_source_info table by @subkanthi in #549
• Replaced slf4j calls with log4j2 api calls by @subkanthi in #553
• Can't load table from Postgres to Clickhouse containing nullable numeric column by @ZlobnyiSerg in #529
• Kafka fixes for multiple database. by @subkanthi in #555
• Added integration test to perform updates on PK to verify incrementin… by @subkanthi in #554
• Enable postgres tests by @subkanthi in #556
• Grafana - Fix prometheus targets by @subkanthi in #558
• Fixed logic of creating sequence number based on debezium timestamp, … by @subkanthi in #557
• Removed excessive logging statements by @subkanthi in #565
• Changed CREATE VIEW to CREATE VIEW IF NOT EXISTS by @subkanthi in #567
• Fix alter drop column by @subkanthi in #560
• Force RMT to old version for Integration tests by @subkanthi in #571
• Changed from ts_ms to debezium ts_ms for adding sequence numbers by @subkanthi in #569
• Fixed alter table change column not null DDL query by @subkanthi in #573
• Changes to make sure the threads are exited when the CLI stop command… by @subkanthi in #525
• remove old broken tests by @Selfeer in #585

New Contributors

• @ZlobnyiSerg made their first contribution in #529

Full Changelog: 2.0.2...2.1.0

Release v2.0.2

Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.2-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-lt/images/sha256-1c0db9877331a0aaceb2e0f2838db3a5156a3cfbfdb83a92235b6287b576c5d0?context=explore

Kafka
Docker: altinity/clickhouse-sink-connector:2.0.2-kafka
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.2-kafka/images/sha256-66e786bbd1230c26abdd0bad70b549abdb1040eab02a25b41c7c7217f51cf0bf?context=explore

What's Changed

• Added release notes for 2.0.1 by @subkanthi in #504
• Update README.md with latest release 2.0.1 by @subkanthi in #505
• 506 datetime limits were updated in jdbc libraryclickhouse java by @subkanthi in #508
• Update config.yml to set database.connectionTimezone by @subkanthi in #514
• Removed log4j 1.x dependencies by @subkanthi in #509
• Updated troubleshooting documentation by @subkanthi in #517
• Added sample application to create 5000 sample records. by @DougTidwell in #521
• Various fixes to the MySQL python commands by @aadant in #515
• ReplicatedReplacingmergetree support for kafka by @subkanthi in #522
• When MAX_QUEUE_SIZE is configured, the this.records.add(batch) method may throw an IllegalStateException and terminate the task. by @UsenPang in #512
• Release 2.0.2 by @subkanthi in #510

New Contributors

• @DougTidwell made their first contribution in #521
• @UsenPang made their first contribution in #512

Full Changelog: 2.0.1...2.0.2

Release v2.0.1

Lightweight
Docker: altinity/clickhouse-sink-connector:2.0.1-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.1-lt/images/sha256-99f38d1b5c34ac8bbcc86a86ca1eadca34e984c77ff1e2b3a54d5ee612029037?context=explore

What's Changed

• Updated README with 2.0.0 release assets by @subkanthi in #481
• 482 rearrange main readme documentation by @subkanthi in #483
• Doc updates for testflows by @Selfeer in #485
• update requirements by @Selfeer in #486
• Update documents for testflows by @Selfeer in #496
• Changed from ConcurrentLinkedQueue to LinkedBlockedQueue with the size configurable by @subkanthi in #491
• Added logic to auto create RRMT tables by @subkanthi in #492
• Separate testflows requirements by @Selfeer in #498
• update requirements by @Selfeer in #499
• update requirements by @Selfeer in #500
• Fix integration tests by @subkanthi in #502
• Release 2.0.1 by @subkanthi in #497

Known issues

• Error running DDL Query - when trying to create a new ReplicatedReplacingMergeTree table #503

Release v2.0.0

Lightweight

Docker: altinity/clickhouse-sink-connector:2.0.0-lt
https://hub.docker.com/layers/altinity/clickhouse-sink-connector/2.0.0-lt/images/sha256-62dee0cae362de871dc6daedef0e705174c0700e903af35299076aae2ab400cb?context=explore

Issues Closed

• #472: Fix old testflows tests
• #471: Testflows datetime tests fix
• #470: MySQL DATETIME values under year 100 are converted incorrectly to ClickHouse's DateTime64.
• #469: Testflows add sink logs
• #467: Testflows test updates
• #465: Add and update TestFlows Lightweight tests
• #464: Integration tests
• #463: the sink-connector lightweight should retry on failures
• #462: Creating a Table With PRIMARY KEY in MySQL With DATETIME Column Results in a Wrong Precision Value in ClickHouse
• #460: Update testflows tests
• #459: Values of the generated columns in MySQL are not transfered to the ClickHouse table
• #457: Testflows ci cd trigger
• #455: Testflows tests for renaming tables and is_deleted
• #454: Update testflows-sink-connector-lightweight.yml
• #453: Refactor DbWriter and DBKafkaOffsetWriter tests
• #452: Add junit test report in github actions
• #451: Constraints are not replicated on table creation but are replicated on new additions
• #450: Fix ddl parser bugs
• #449: Update README.md
• #448: Total records in one table doubled when i just update or create new one record
• #444: Update README.md
• #443: Added logic to store the record committer object and to acknowledge t…
• #442: Removed log4j 1.x and other unused dependencies
• #439: Added logic in auto.create.table to support table names with dash(hyphen)
• #438: new table in Postgresql
• #436: Added logic to launch debezium event loop in a separate thread, set t…
• #435: renaming tables when copying from postgres
• #434: Rename is_deleted column if source tables have column with the same name
• #433: 429 support for calculated columns(mysql generate)
• #432: Add clickhouse jdbc parameter config
• #431: Support source tables with is_deleted columns
• #430: Implement a fast table row count
• #429: Support for calculated columns
• #428: Add ability to pass JDBC parameters as configuration variables
• #427: Fix unjustified xfails
• #426: Add a python wrapper for mysqlsh
• #425: Refactored the overriden functions in Antlr parser to a separate clas…
• #422: CREATE TABLE TIMESTAMP PRECISION IS NOT MAPPED CORRECTLY to DATETIME64
• #420: [Fix] Export CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE instead of SINK_LIGHTWEIGHT_VERSION
• #419: [Fix] add registry to the clickhouse_debezium_embedded
• #418: Ability to control the maximum batch size inserted to ClickHouse
• #417: Alter table add/drop column PostgreSql
• #416: Add --max_datetime_value to compare with CH DateTime64
• #414: Updated debezium version to 2.5.0.Beta1
• #413: sink connector cannot insert data into table named transaction
• #412: Allow for password less command lines
• #410: Update docker-compose-mysql.yml
• #409: Update README.md to include docker hub path
• #408: Change rest api parameter for binlog_position to match client
• #407: Added Integration test for testing debezium offset storage, fixed postgres yaml file
• #405: Added comments to sink connector lightweight configuration
• #404: Rework docker-compose-mysql.yml example used in the quikcstart guide
• #402: Restructure README and linked docs
• #399: Docker tag in Java tests
• #398: clickhouse-loader.py and *checksum.py security
• #396: PR from another fork
• #395: upload jar artifacts
• #394: Myro tk patch 1
• #393: Sanity run
• #392: Update docker upload, release pipeline, update testflows tests
• #391: Added step in docker-build.yml github workflow to set sink connector …
• #390: Add github step in workflow to update sink-connector jar version in lightweight pom.xml
• #389: Removed doc, docs folder, cleanup docker compose and update titles of…
• #388: Clean up documentation
• #387: Test PR from another fork
• #386: Configuration variable to disable replication.
• #383: Test PR from a different fork
• #382: Pull request workflow
• #381: Docker tag as environment variable
• #380: The sink-connector lightweight stops replicating
• #379: Fix datetime limits
• #378: testflow tests docker tag not using the version specified in the workflow file
• #376: Added tests to validate DATETIME values from datatypes.sql
• #375: Fix nullable fields and add clickhouse_datatime_timezone
• #374: Param source timezone (review)
• #373: Add logic to limit DateTime max
• #372: Update instructions to connect to cloud url.
• #367: Insert fails when its a DateTime64 column with timezone
• #366: Added additional test cases for version check and MergeTree tables, Add backticks to column names(fix for names with spaces))
• #362: Escape column names using backticks
• #361: Update README.md
• #358: Python utility clickhouse_table_checksum fails for data_types database
• #356: Updated Debezium to 2.5.0.Alpha1(Latest release), Added mariadb docke…
• #355: Update README.md
• #353: Config variable to start the sink-connector without turning on replication
• #351: Include minimal description of variables in the config.yml file
• #349: Specify timezone for DateTime / DateTime64
• #347: Debezium offset configuration variables were updated which breaks offset storage
• #346: Fix integration tests
• #344: Fix for UNKNOWN DATABASE exception
• #340: Choose the binary data type encoding in the sink-connector
• #338: clickhouse_loader.py : fixing temporal and binary data types
• #332: Support partitions when migrating to CH
• #328: Correctly handle temporal type Time zone.
• #326: Atomic update of the altinity_sink_connector.replica_source_info
• #321: CREATE TABLE DDL not transformed properly
• #319: java.util.ConcurrentModificationException from the DbWriter
• #290: Use debezium version 2.4
• #253: Better exception handling when STRUCT is empty
• #232: DateTime64 conversion logic
• #230: Offset acknowledgment - Add logic to persist/dump offset only after records are persisted to ClickHouse
• #193: Error parsing Gtid when values exceeds MAX INTEGER
• #126: 122 remove trailing zeros in timestamp and time columns
• #124: testflows & github action tests
• #85: Create row policy on auto create table to hide deleted rows
• #60: Manually set binlog position before starting streaming.
• #52: Kubernetes updates - Added redpanda 3 node cluster configuration and schema registry confl…
• #49: Document steps to connect to external MySQL server and instructions to load ontime dataset.
• #28: Support for JSON data type.
• #23: Map INT32 Kafka connect to INT256 CH type
• #16: Create sysbench script combine tests
• #14: Refactor logic to buffer records and flush.
• #1: Added initial commit of files from the original repo

Pull Requests Merged

• #464: Integration tests
• #472: Fix old testflows tests
• #471: Testflows datetime tests fix
• #469: Testflows add sink logs
• #450: Fix ddl parser bugs
• #467: Testflows test updates
• #465: Add and update TestFlows Lightweight tests
• #443: Added logic to store the record committer object and to acknowledge t…
• #460: Update testflows tests
• #457: Testflows ci cd trigger
• #455: Testflows tests for renaming tables and is_deleted
• #454: Update testflows-sink-connector-lightweight.yml
• #449: Update README.md
• #436: Added logic to launch debezium event loop in a separate thread, set t…
• #444: Update README.md
• #442: Removed log4j 1.x and other unused dependencies
• #439: Added logic in auto.create.table to support table names with dash(hyphen)
• #374: Param source timezone (review)
• #434: Rename is_deleted column if source tables have column with the same name
• #433: 429 support for calculated columns(mysql generate)
• #430: Implement a fast table row count
• #425: Refactored the overriden functions in Antlr parser to a separate clas…
• #432: Add clickhouse jdbc parameter config
• #426: Add a python wrapper for mysqlsh
• #405: Added comments to sink connector lightweight configuration
• #427: Fix unjustified xfails
• #414: Updated debezium version to 2.5.0.Beta1
• #420: [Fix] Export CLICKHOUSE_SINK_CONNECTOR_LT_IMAGE instead of SINK_LIGHTWEIGHT_VERSION
• #419: [Fix] add registry to the clickhouse_debezium_embedded
• #416: Add --max_datetime_value to compare with CH DateTime64
• #404: Rework docker-compose-mysql.yml example used in the quikcstart guide
• #408: Change rest api parameter for binlog_position to match client
• #412: Allow for password less command lines
• #410: Update docker-compose-mysql.yml
• #409: Update README.md to include docker hub path
• #407: Added Integration test for testing debezium offset storage, fixed postgres yaml file
• #379: Fix datetime limits
• #402: Restructure README and linked docs
• #399: Docker tag in Java tests
• #396: PR from another fork
• #394: Myro tk patch 1
• #393: Sanity run
• #381: Docker tag as environment variable
• #395: upload jar artifacts
• #392: Update docker upload, release pipeline, update testflows tests
• #391: Added step in docker-build.yml github workflow to set sink connector …
• #387: Test PR from another fork
• #389: Removed doc, docs folder, cleanup docker compose and update titles of…
• #382: Pull request workflow
• #383: Test PR from a different fork
• #376: Added tests to validate DATETIME values from datatypes.sql
• #375: Fix nullable fields and add clickhouse_datatime_timezone
• #366: Added additional test cases for version check and MergeTree tables, Add backticks to column names(fix for names with spaces))
• #356: Updated Debezium to 2.5.0.Alpha1(Latest release), Added mariadb docke…
• #372: Update instructions to connect to cloud url.
• #361: Update README.md
• #338: clickhouse_loader.py : fixing temporal and binary data types
• #344: Fix for UNKNOWN DATABASE exception
• #355: Update README.md
• #346: Fix integration tests

Kafka version with support for passing JDBC parameters.

Kafka version with support for passing JDBC parameters.
Docker: altinity/clickhouse-sink-connector:latest

Sink connector Kafka(added binary encoding option)

Added option to write UUID/BLOB(Debezium.BYTES) datatype as raw bytes instead of encoding as Base64 or HEX strings.

pre_release_1.0.0

Config.yml needs to be updated
From offset.storage.offset.storage.jdbc.offset.table.name to offset.storage.jdbc.offset.table.name
From offset.storage.offset.storage.jdbc.offset.table.ddl to offset.storage.jdbc.offset.table.ddl

• Fix for gtid overflow integer error.

clickhouse-debezium-embedded-1.1-SNAPSHOT.jar

Support for Postgres data types(Array)

What's Changed

• Added logic to ignore deletes by @subkanthi in #282
• Fix kakfa metadata in Kafka sink connector by @subkanthi in #286
• Docusarus docs by @subkanthi in #283
• Added support for postgresql data types(including Array). by @subkanthi in #297
• 296 postgresql data types add test cases and support by @subkanthi in #298

Full Changelog: release_0.7.0...release_0.9.0