Releases: snowflakedb/snowflake-kafka-connector
v3.0.0
Notable Changes
- Kafka Connector can ingest data into an Iceberg table with Snowpipe Streaming
** Please refer to documentation for detailed information on setup and known limitations - Changes in default parameter values
** is enabled by default for Snowpipe Streaming ingestion
** default value is set to 120 seconds to match the default value of . - Fixed dependency vulnerabilities
What's Changed
- NO-SNOW Change kafka-connect-avro-converter scope to provided by @sfc-gh-mbobowski in #985
- NO-SNOW Remove FIPS test by @sfc-gh-mbobowski in #989
- No snow use com.fasterxml json by @sfc-gh-bzabek in #994
- SNOW-1728002 get rid of reflection when enabling iceberg streaming by @sfc-gh-bzabek in #995
- SNOW-1665420 add logic to parse Iceberg schema by @sfc-gh-bzabek in #996
- NO-SNOW Enable IcebergInitServiceIT by @sfc-gh-mbobowski in #1001
- SNOW-1728000 Iceberg e2e tests by @sfc-gh-mbobowski in #1002
- SNOW-1761519 Fix empty arrays in Iceberg ingestion by @sfc-gh-mbobowski in #1003
- [Snyk] Fix for 1 vulnerabilities by @sfc-gh-snowflakedb-snyk-sa in #1004
- NO_SNOW Update Snowflake JDBC to 3.20.0 by @sfc-gh-xhuang in #1000
- SNOW-1805174 Enable single buffer by default by @sfc-gh-mbobowski in #1005
- Update ConnectorConfigDefinition.java by @sfc-gh-achyzy in #1009
- SNOW-1831140 Iceberg schema evolution e2e test setup by @sfc-gh-mbobowski in #1010
- NO-SNOW Fix missing ubuntu packages error in CI by @sfc-gh-mbobowski in #1011
- SNOW-1729292 modify iceberg tree based on record data by @sfc-gh-bzabek in #1007
- SNOW-1831140 E2e tests of Iceberg JSON ingestion by @sfc-gh-mbobowski in #1013
- SNOW-1842220 Add comments to columns created by schema evolution by @sfc-gh-bzabek in #1014
- SNOW-1831140 Complete Iceberg e2e tests by @sfc-gh-mbobowski in #1015
- NO-SNOW Update Confluent dependencies to 7.7.2 by @sfc-gh-mbobowski in #1019
- SNOW-1820302 Integrate with ingest sdk 3.0.1 by @sfc-gh-mbobowski in #1016
- NO-SNOW Update jackson and commons-compress by @sfc-gh-mbobowski in #1021
- SNOW-1731255: Put null or empty json nodes to DLQ for schema evolution by @sfc-gh-wtrefon in #1017
- NO-SNOW Release 3.0.0 by @sfc-gh-mbobowski in #1022
Full Changelog: v2.5.0...asdasd
Contributors
Assets 2
v2.5.0
Notable Changes
- Updated Streaming Ingest SDK to the 2.3.0 version
- Closing channels in parallel is now enabled by default. It reduces the time needed for the connector restart.
What's Changed
- SNOW-1658905 Limit SnowflakeSinkConnectorConfig responsibilities(#925)
- NO-SNOW Enable closing channels in parallel by default(#928)
- SNOW-1680410 adjust validations and create metadata column if not exists(#946)
- SNOW-1278872 Added jdk version and vendor to metrics(#972)
- Fix logging for closing partition channels - partition and topic misplaced(#977)
v2.4.1
Notable Changes
- Updated Streaming Ingest SDK to the 2.2.2 version
What Changed
SNOW-1649161 Fix NaN value handling in schematization (#920)
SNOW-1623269 Fail sink task on authorization exception from Snowflake (#916)
SNOW-1514185 Assign new channel when no offset is present in Snowflake (#913)
SNOW-1571459 parse jbdc properties (#909)
v2.4.0
Notable changes
• Upgrade of Snowflake Ingest SDK to the latest 2.2.0 version - containing a critical fix for potential corruption when change_tracking is enabled. - see https://github.com/snowflakedb/snowflake-ingest-java/releases/tag/v2.2.0 for more details
We strongly recommend upgrading Kafka connector to this 2.4.0+ version if you are using Snowpipe Streaming functionality
• Upgrade of Snowflake JDBC driver from version 3.14.5 to version 3.18.0 - see https://docs.snowflake.com/en/release-notes/clients-drivers/jdbc-2024 for more details,
• Fixed issue with connector loosing data when reopening a channel fails before fully initializing channel: Fix to SNOW-1514185: Do assign the reopened channel unless Kafka offsets are fully reset (#875),
• Improved logging experience in various components for improved troubleshooting experience,
• Updated dependencies with known vulnerabilities.
What's Changed
• SNOW-1618257 - upgrade to 2.2.0 ingest-sdk by @sfc-gh-xhuang in (#910)
• NO_SNOW Update dependencies by @sfc-gh-xhuang in (#898)
• SNOW-1229108 Add warning about buffer config when single buffer enabled by @sfc-gh-mbobowski in (#896)
• NO-SNOW Ignore OAuth test by @sfc-gh-mbobowski in (#906)
• Bump org.sonatype.plugins:nexus-staging-maven-plugin from 1.6.7 to 1.7.0 in (#857)
• Bump io.dropwizard.metrics:metrics-core from 4.2.3 to 4.2.26 in (#863)
• Bump org.assertj:assertj-core from 3.25.3 to 3.26.3 (#881)
• Bump dev.failsafe:failsafe from 3.2.1 to 3.3.2 (#859)
• SNOW-947731 Remove deprecated avro-python3 package by @sfc-gh-mbobowski in (#889) (
• SNOW-1541942 Extend Snowpipe initialization logging at INFO level by @sfc-gh-mbobowski in (#885)
• SNOW-1229110 Add single buffer usage data to telemetry by @sfc-gh-mbobowski in (#893)
• Upgrade com.google.protobuf to 3.24.4 by @sfc-gh-wtrefon in (#892)
• NO-SNOW Upgrade nexus-staging-maven-plugin version by @sfc-gh-xhuang in (#887)
• Remove insertRows DEBUG log (called once per row) by @sfc-gh-mbobowski in (#886)
• Adding logs to track schematization by @sfc-gh-mbobowski in (#884)
• NOSNOW: improve troubleshooting experience for scenario with skipped offsets by @sfc-gh-gjachimko in (#883)
• SNOW-1061855 Fix the E2E test for SchemaEvolutionDropTable with a single buffer by @sfc-gh-gjachimko in (#882)
• SNOW-1061851: Run SnowflakeSinkServiceV2IT for a single buffer by @sfc-gh-akowalczyk in (#876)
• SNOW-1514185: Do assign the reopened channel unless kafka offsets are fully reset by @sfc-gh-akowalczyk in (#875)
• NOSNOW: added utility to upload connector to our nexus for simplified k8s deployment by @sfc-gh-gjachimko in (#874)
Contributors
Assets 2
v2.3.0
Notable changes
- (Snowpipe Streaming) Timestamp of pushing data into Snowflake is added to record_metadata. Note according to the configured kafka buffers, the timestamp value may be have a slight delay between when it is pushed to the Snowpipe Streaming channel buffer and when it is flushed and available on the table.
- SNOW-1056407 Add SnowflakeConnectorPushTime to record_metadata by @sfc-gh-lshcharbaty in #833
- (Snowpipe Streaming) Optional parameter,
snowflake.streaming.closeChannelsInParallel.enabled, for Kafka connector to close channels in parallel which significantly speeds up rebalance time- SNOW-1369280 Close channels asynchronously by @sfc-gh-lshcharbaty in #841
- Updated dependencies with known vulnerabilities
- SGRC-4405 Update Kafka to 3.7.0 and Confluent to 7.6.0 by @sfc-gh-mbobowski in #854
What's Changed
- SNOW-1269024 Installing multiple Java versions for test purposes by @sfc-gh-krosinski in #824
- SNOW-1269025 Testing Confluent kafka connector using multiple JRE versions by @sfc-gh-krosinski in #826
- SNOW-1269038 Stress testing Confluent kafka connector using multiple JRE versions by @sfc-gh-krosinski in #827
- SNOW-1348656 Migrate RecordContentTest to JUnit5 by @sfc-gh-mbobowski in #828
- SNOW-1056407 Cleanup metadata properties class. Migrate records tests to junit5 by @sfc-gh-lshcharbaty in #831
- NO-SNOW Fix build warnings by declaring maven-plugin versions by @sfc-gh-xhuang in #837
- NO-SNOW Fix idempotent multiple column schema evolution by @sfc-gh-xhuang with contribution from @cmonty-paypal in #836
- SNOW-1348656 Send records that could not be parsed to DLQ by @sfc-gh-mbobowski in #830
- SNOW-1056407 Add tests MetaColumnTest for snowpipe streaming by @sfc-gh-lshcharbaty in #832
- SNOW-1369271 Ignore SFException when closing a channel by @sfc-gh-lshcharbaty in #839
- NO-SNOW Remove unused code path from RecordService by @sfc-gh-mbobowski in #838
- NO-SNOW Ugrade to ingest-sdk 2.1.1 by @sfc-gh-xhuang in #843
- SNOW-1167007: Migrate schema when receiving null values for not null cols by @sfc-gh-wtrefon in #849
- SNOW-1061848: Introduce TopicPartitionChannel interface by @sfc-gh-akowalczyk in #851
- SNOW-1061822: Add new properties for streaming client by @sfc-gh-wtrefon in #822
- SNOW-1061848: DirectTopicPartitionChannel - copy of the old implementation with removed buffer by @sfc-gh-akowalczyk in #852
- SNOW-1061848: Fix unit tests by @sfc-gh-akowalczyk in #855
- NO-SNOW Add dependabot by @sfc-gh-mbobowski in #856
- SNOW-1061848: Run TopicPartitionChannelIT against a non-buffered path by @sfc-gh-akowalczyk in #862
- SNOW-1475664: Move additional error handling from BufferedTopicPartitionChannel to fix E2E test by @sfc-gh-akowalczyk in #865
- NO_SNOW Add column comments from schemaregistry by @sfc-gh-akowalczyk with contribution from @wornjs in #870
- SNOW-1061854 Run e2e tests with single buffer by @sfc-gh-lshcharbaty in #867
- SNOW-1019628 Enable V2 file cleaner for Snowpipe ingestion by default by @sfc-gh-gjachimko in #868
- Release v2.3.0 by @sfc-gh-mbobowski in #873
New Contributors
- @sfc-gh-krosinski made their first contribution in #824
- @sfc-gh-nsienkiewicz-1 made their first contribution in #842
Full Changelog: v2.2.2...v.2.3.0
Contributors
Assets 2
v2.2.2
Notable changes
- Fixed bug that caused stage files being not deleted in Snowpipe ingestion
- SNOW-1019628 update file cleaner for snowpipe ingest by @sfc-gh-gjachimko in #807
What's Changed
- SNOW-1055524 E2E test with nullable SMT by @sfc-gh-lshcharbaty in #808
- SNOW-1057165 - Test-doubles for snowflake-ingest-java by @sfc-gh-achyzy in #809
- SNOW-1055524 E2E SMT without schema evolution by @sfc-gh-lshcharbaty in #811
- SNOW-1055524 Check table schema in e2e smt by @sfc-gh-lshcharbaty in #813
- SNOW-1055524 E2E SMT for a Snowpipe based connector by @sfc-gh-lshcharbaty in #812
- NO-SNOW Remove performance tests by @sfc-gh-mbobowski in #817
- SNOW-1055561: Check whether SMT returning null values no longer stops a data ingestion. by @sfc-gh-akowalczyk in #816
- Update to release 2.2.2 by @sfc-gh-rcheng in #819
New Contributors
- @sfc-gh-lshcharbaty made their first contribution in #808
- @sfc-gh-achyzy made their first contribution in #809
- @sfc-gh-akowalczyk made their first contribution in #816
- @sfc-gh-krosinski made their first contribution in #824
Full Changelog: v2.2.1...v2.2.2
Contributors
Assets 2
v2.2.1
Notable Changes
- SNOW-979849: Add offset verification logic by @sfc-gh-tzhang in #795
- SNOW-1117446 Snowpipe Streaming client provider map by @sfc-gh-japatel in #794
- SNOW-1049869 Cleanup streaming ingest threads when SinkTask stop() is called by @sfc-gh-mbobowski in #800
What's Changed
- SNOW-922795 Added third-party-licenses to .zip distribution by @sfc-gh-mbobowski in #784
- PROD-39429 Add PR template by @sfc-gh-japatel in #786
- Run subset of e2e tests by @sfc-gh-mbobowski in #788
- NO-SNOW: upgrade JDBC to 3.14.5 by @sfc-gh-tzhang in #799
- [external contributor] Upgrade dependency versions by @bansallovish merged by @sfc-gh-xhuang in #802
- Upgrade connect-api to 3.7.0 by @sfc-gh-mbobowski in #803
- SNOW-1057151: Add poc test setup for EmbeddedConnectCluster by @sfc-gh-wtrefon in #801
- SNOW-1049869 Rewrite client optimization unit tests by @sfc-gh-mbobowski in #804
- Release v2.2.1 by @sfc-gh-rcheng in #805
New Contributors
- @sfc-gh-mbobowski made their first contribution in #784
- @sfc-gh-wtrefon made their first contribution in #801
Full Changelog: v2.2.0...v2.2.1
Contributors
Assets 2
v2.2.0
Notable Changes
- Security fix that enforces client authentication that results in more clients being created. This may increase your client usage costs
- [SNOW-954150] Use map of clients with different configurations instead of one client for multiple connectors configurations by @sfc-gh-rcheng in #744
- Corruption fix that ensures exactly once delivery for schema evolution edge cases
- SNOW-943288, SNOW-938379: Fix two schema evolution cases that could cause non-exactly once delivery by @sfc-gh-tzhang in #775
What's Changed
- NO-SNOW Make schema evolution add columns idempotent by @sfc-gh-xhuang in #734
- NO-SNOW: Preserve the old data type that goes into an ARRAY column for Schematization by @sfc-gh-tzhang in #730
- SNOW-938038: Support AVRO Logical Types by @sfc-gh-tzhang in #722
- [SNOW-943288] Do not skip records when we're expecting the offset to be reset by @sfc-gh-rcheng in #729
- PROD-39429 Implement migrate sys func from new channel(Format V2) to old channel (V1) - Push to Main by @sfc-gh-japatel in #751
- [external contributor] Support config providers in validation by @swehner by @sfc-gh-rcheng in #753
- NO-SNOW: Expose the Ingest SDK MAX_CLIENT_LAG configuration in KC by @sfc-gh-tzhang in #758
- Add authentication for max lag parameter and fix IT by @sfc-gh-rcheng in #763
- SNOW-989387: Fix for java.lang.NoClassDefFoundError by @sfc-gh-wfateem in #774
- [SNOW-1015644] Update ingest sdk to 2.0.5 by @sfc-gh-rcheng in https://github.com/snowflakedb/snowflake-kafka-connector/pull/778f
- resolves issue with generating and building java libary by @sfc-gh-gjachimko in #780
- Update version to 2.2.0 for release by @sfc-gh-rcheng in #779
- Downgrade JDBC to 3.13.30 by @sfc-gh-rcheng in #783
New Contributors
- @sfc-gh-xhuang made their first contribution in #734
- @sfc-gh-gjachimko made their first contribution in #780
Full Changelog: v2.1.2...v2.2.0
Contributors
Assets 2
v2.1.2
IMPORTANT This version supplants previous 2.1.x versions and is the recommended version. Do not use earlier 2.1.x versions.
What's Changed
- SNOW-189106 Kafka Connector to Support External OAuth by @sfc-gh-alhuang in #671
- [SNOW-870373] Enable JMX metrics for Snowpipe Streaming by @sfc-gh-rcheng in #674
- [SNOW-851840] Enable tombstone record ingestion in Snowpipe Streaming by @sfc-gh-rcheng in #688, #709 and #700
- SNOW-903979 Set default to true for one client optimization by @sfc-gh-japatel in #695
- SNOW-808908: Handle Reserved Keywords and Special Characters in Schematization by @sfc-gh-tzhang in #693
- [Streaming Telemetry 2][SNOW-899866] Enables reportKafkaPartitionStart/Usage telemetry for Streaming by @sfc-gh-rcheng in #694 and #696
- [NO-SNOW] Prefix with channel name with connector name by @sfc-gh-japatel in #703
- PROD-39429 Implement migrate sys func from new channel(Format V2) to old channel (V1) by @sfc-gh-japatel in #750
Package Updates
- Ingest SDK 2.0.2 to 2.0.3 by @sfc-gh-rcheng in #699
- Avro 1.11.1 to 1.11.3 by @sfc-gh-snowflakedb-snyk-sa in #713 and #718
New Contributors
- @sfc-gh-alhuang made their first contribution in #683
- @sfc-gh-snowflakedb-snyk-sa made their first contribution in #713
Full Changelog: v2.0.1...v2.1.2
Contributors
Assets 2
2.0.2-rc
This release is on top of release https://github.com/snowflakedb/snowflake-kafka-connector/releases/tag/v2.0.1
Important commits between two are: