v2.0.1

Notable Changes

• SNOW-811265 Not clearing entire map of partitionToChannel, only remove by @sfc-gh-japatel in #687
• SNOW-858328 Add cache for schema evolution permission query by @sfc-gh-alhuang in #683

Other Changes

• Small changes to Snowpipe Telemetry and JMX by @sfc-gh-rcheng in #672
• Change pipe already registered log from error to warn level by @sfc-gh-rcheng in #679

Full Changelog: v2.0.0...v2.0.1

v2.0.0

Snowpipe Streaming with Kafka Connector is now Generally Available

Reverts c5bb312 to close channel on rebalance

Full Changelog: v1.9.4...v2.0.0

1.9.4

One client

• Release 1.9.4 introduces the enable.streaming.client.optimization option, which is enabled by default.
• With this client optimization, only one client is created for multiple topic partitions per Kafka connector. This feature can reduce client runtime and lower migration cost by creating larger files.
• Note that in a high throughput scenario (for example, 50 MB/s per connector), we recommend that you disable this property if you see an increase in latency or costs.

What's Changed

• Refactor buffer threshold variables and methods to be more consistent by @sfc-gh-rcheng in #639
• [SNOW-840882] Disable flakey streaming client test by @sfc-gh-rcheng in #654
• SNOW-826772 Unification of Snowflake role and user for Snowpipe Streaming by @sfc-gh-japatel in #650
• NO-SNOW Kafka connector supports any encoding, not only UTF-8 by @sfc-gh-japatel in #652
• Upgrade to SDK 2.0.1 and fix corresponding override bdec version test by @sfc-gh-rcheng in #653
• no-snow update Google guava by @sfc-gh-japatel in #655
• enzo-cappa fix: add warn log when a file failed to be ingested by @sfc-gh-rcheng in #656
• Revert removal of TestSchemaEvolutionJson e2e test by @sfc-gh-rcheng in #659
• SNOW-811265 Do not close channel on rebalance by @sfc-gh-japatel in #651
• SNOW-850959: Fix a wrong result issue that offsets are skipped when schematization is enabled by @sfc-gh-tzhang in #658
• [SNOW-857447] Enable one client optimization by default by @sfc-gh-rcheng in #665
• Release version 1.9.4 by @sfc-gh-rcheng in #666

Changes made but reverted

• Add, attempt fix and ultimately remove flakey CLA bot
  • Fix cla bot by using correct github token by @sfc-gh-rcheng in #640
  • Remove CLA bot and update README.md to include cla and formatter requirements by @sfc-gh-rcheng in #643
• Change Kafka put and precommit log level to info and then revert
  • Revert kafka put and precommit log message to info by @sfc-gh-rcheng in #637
  • NO-SNOW Too many info logs for successfully called put and precommit,… by @sfc-gh-japatel in #660

Full Changelog: v1.9.3...v1.9.4

1.9.3

What's Changed

• SNOW-791549: Infer NULL to VARCHAR instead of VARIANT since ingestion timestamp into VARIANT column is not allowed by @sfc-gh-tzhang in #625
• Use MDC context instead of kc and logger task by @sfc-gh-rcheng in #568
• NO-SNOW: fix commit offset behavior for Snowpipe by @sfc-gh-tzhang in #631
• One client change 2/3: Add client provider by @sfc-gh-rcheng in #590
• One client change 3/3: Call into client provider and corresponding resilience tests by @sfc-gh-rcheng in #609
• Enable topic regexes for table2topic mapping by @sfc-gh-rcheng in #628
• SNOW-480523 Fix Downscope URL for GCS by @sfc-gh-japatel in #634

Full Changelog: v1.9.2...v1.9.3

1.9.2

Update:

Do not use version 1.9.2 for Snowpipe based Kafka Connector. Check v1.9.3 to see what bugs are fixed in this version.

What's Changed

• Include bcpkix dependency in Confluent Uber Jar so be able to not add… by @sfc-gh-japatel in #563
• SNOW-758111 Use telemetry service for all exceptions wherever possible by @sfc-gh-japatel in #569
• [NO-SNOW] MINOR - Update README.md by @sfc-gh-japatel in #572
• Disable delivery guarantee tests by @sfc-gh-japatel in #577
• NO-SNOW Best practice: Use fail-safe plugin for IT tests by @sfc-gh-japatel in #578
• SNOW-760171 Use pom_confluent.xml in running confluent tests by @sfc-gh-japatel in #573
• Update README.md Links by @sfc-gh-japatel in #581
• NO-SNOW Use maven Cache - 4X improvement in runtime of Github Actions for builds by @sfc-gh-japatel in #579
• SNOW-760171 Part 1: Test to verify DLQ results by @sfc-gh-japatel in #585
• SNOW-759517, SNOW-743629, SNOW-733467: simplify offset management logic in KC by @sfc-gh-tzhang in #566
• [SNOW-659469] List all invalid parameters by @sfc-gh-rcheng in #586
• SNOW-756851 Part 1: Refactor code to create telemetry service for Snowpipe Streaming by @sfc-gh-japatel in #583
• Reenable accidentally disabled config tests from PR 586 by @sfc-gh-rcheng in #588
• One client change 1/3: add parameter to enable one client optimization by @sfc-gh-rcheng in #589
• SNOW-768488: Accept any offset from kafka if InvalidOffsetException is thrown during startup by @sfc-gh-tzhang in #587
• [SNOW-748118] Remove delivery guarantee config param by @sfc-gh-rcheng in #551
• NO-SNOW Refactor Test suites + Disable failing tests to unblock other PRs by @sfc-gh-japatel in #598
• NO-SNOW Use a different github key to reduce github API limits by @sfc-gh-japatel in #600
• SNOW-480523 Upgrade JDBC to 3.13.25 to use downscope gcs token instea… by @sfc-gh-japatel in #593
• Add cla bot from snowflake-connector-python repo by @sfc-gh-rcheng in #597
• SNOW-774533 Enable Json & Proxy Test - Bug in setting up Confluent Tests by @sfc-gh-japatel in #605
• SNOW-480523 Fix breaking change in upgrade to JDBC 3.13.25 by @sfc-gh-japatel in #607
• Add format script by @sfc-gh-rcheng in #611
• SNOW-774533 Fix DLQ bug, re-enable test by @sfc-gh-japatel in #604
• Add telemetry for fatal errors in TopicPartitionChannel by @sfc-gh-japatel in #601
• NO-SNOW Add caching layer for maven to speed up Github action by @sfc-gh-japatel in #613
• [SNOW-726924] Add New jvm.nonProxy.hosts Parameter - Undo Revert by @sfc-gh-japatel in #614
• Vuln Update: Upgrade JDBC to 3.13.29 by @sfc-gh-japatel in #616
• Add resilience testing by @sfc-gh-rcheng in #617
• SNOW-735650: sfc-jenkins -> sfc-eng-jenkins s3 bucket change by @sfc-gh-sscott in #532
• NO-SNOW Delete Full tests as cron jobs as they are now part of every PR by @sfc-gh-japatel in #621
• NO-SNOW: fix offset token behavior when resetting the offset to data that expired by @sfc-gh-tzhang in #619
• SNOW-803249 Remove confluent 5.5.11 since it is no longer supported by @sfc-gh-japatel in #623
• [NO-SNOW] Use latest version of Ingest SDK by @sfc-gh-japatel in #622
• NO-SNOW Release version 1.9.2 by @sfc-gh-japatel in #624

New Contributors

• @sfc-gh-sscott made their first contribution in #532

Full Changelog: v1.9.1...v1.9.2

1.9.1

What's Changed

• NO-SNOW Revert one client change, JDBC upgrade and config for arrow and parquet by @sfc-gh-japatel in #555
• SNOW-740327 Create config for using Arrow BDEC file format - Redo aft… by @sfc-gh-japatel in #556
• NO-SNOW: allow StringConverter when schematization is in the config but set to false by @sfc-gh-tzhang in #560
• NO-SNOW Release 1.9.1 for Snowpipe Streaming by @sfc-gh-japatel in #559

Full Changelog: v1.9.0...v1.9.1

1.9.0

What's Changed

• This version supports ingestion through Snowpipe Streaming which is the first non-beta version after 1.6.9, it uses v1.1.0 of Ingest SDK as the dependency which is also non-beta.
• All 1.8.x versions uses older version of Ingest SDK.
• It is advised to use v1.9.x going forward if you are planning to use Snowpipe Streaming.
• No changes if Snowpipe is still the preferred way of ingestion.

Major Changes

• Bump Ingest SDK to 1.1.0 and release 1.9.0 by @sfc-gh-japatel in #552
• [SNOW-692657] Create one client per connector instead of one client per task by @sfc-gh-rcheng in #528 - The change is not rolled out through a parameter.

Minor Changes

• NO-SNOW: match pom_confluent.xml with pom.xml by @sfc-gh-tzhang in #531
• [SNOW-726924] Add New jvm.nonProxy.hosts Parameter and update JDBC to 3.13.23 by @sfc-gh-wfateem in #533
• Fix client off by one issue by @sfc-gh-rcheng in #540
• Refactor sink task integration tests to be more readable by @sfc-gh-rcheng in #543
• [Snyk] Security upgrade org.apache.kafka:connect-api from 3.2.3 to 3.4.0 by @snyk-bot in #539
• Modify pom.xml to use jdbc defined in connector and not from ingest sdk by @sfc-gh-japatel in #546
• SNOW-740327 Create config for using Arrow BDEC file format + Bump Client SDK 1.0.3-beta by @sfc-gh-japatel in #541
• no-snow Add connector name in KC client name for better debugging the client … by @sfc-gh-japatel in #547

New Contributors

• @sfc-gh-wfateem made their first contribution in #533
• @snyk-bot made their first contribution in #539

Full Changelog: v1.6.9...v1.9.0

v1.8.2

What's Changed

• SNOW-645776 Part 2 Stress Tests run on Confluent's Different Versions Against Different Cloud by @sfc-gh-japatel in #478
• NO-SNOW Fix release issue since version upgrades were not applied to confluen… by @sfc-gh-japatel in #482
• Add docker setup resources to the kafka connect repo by @sfc-gh-rcheng in #483
• Fix docker-setup yml files by @sfc-gh-rcheng in #490
• SNOW-625820: multiple fixes for KC schematization schema mapping by @sfc-gh-tzhang in #488
• [SNOW-594195] Add correlationId to logging by @sfc-gh-rcheng in #487
• NO-SNOW: Put rows with JsonProcessingException into the DLQ instead of ignoring them by @sfc-gh-tzhang in #491
• [SNOW-643653] Fix Citi Blackduck vulnerabilities by @sfc-gh-rcheng in #497
• [SNOW-665920] Add warnings in build script to prevent a privately built jar from going to the customer by @sfc-gh-rcheng in #492
• [SNOW-665898] Add log granularity instance id logging for Task by @sfc-gh-rcheng in #495
• Modify Codeowners by @sfc-gh-japatel in #494
• SNOW-630885: Support schema evolution with schematization by @sfc-gh-tzhang in #501
• Bump protobuf-java from 3.19.4 to 3.19.6 in /test/test_data/protobuf by @dependabot in #499
• SNOW-686528: Check schema evolution table option by @sfc-gh-tzhang in #509
• SNOW-506446: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.13.2.1 to 2.13.4.2 by @sfc-gh-snyk-sca-sa in #503

New Contributors

• @sfc-gh-snyk-sca-sa made their first contribution in #503

Full Changelog: v1.8.1...v1.8.2

1.6.9

Update packages to fix security vulnerabilities

What's Changed

• Fix security vulnerabilities and upgrade KC version to 1.6.9 by @sfc-gh-rcheng in #498

Full Changelog: v1.6.8...v1.6.9

1.8.1

Major Commits

• SNOW-514303 Support n:1 topics:table by @sfc-gh-zefan in #459
• Upgrade Ingest SDK to 1.0.2-beta.4 for KC release 1.8.1 by @sfc-gh-japatel in #480 (Check release details https://github.com/snowflakedb/snowflake-ingest-java/releases/tag/v1.0.2-beta.4)
• [BUG_FIX] Fix for invalid connector name - Issue 456 by @sfc-gh-japatel in #465
• SNOW-625820 Schema mapping by @sfc-gh-zefan in #467 (Preview)
• SNOW-628677 Auto Table Creation by @sfc-gh-zefan in #470 (Preview)
• Support for Kafka connect API - 3.2.0
  • Please note, this is provided scope. Since users of this plugin use their own kafka connect and broker runtime versions, it has nothing to do with this kafka connect API version. This upgrade only means we are compatible and tested with this version.

All Commits

• PRODSEC-1263 Whitesource Transition to Snyk by @sfc-gh-jfan in #433
• NO-SNOW: Fix two issues in Kafka Connector by @sfc-gh-tzhang in #463
• NO-SNOW Rotate private keys for all three sfctest0 accounts by @sfc-gh-japatel in #464
• Release version 1.8.1 by @sfc-gh-tzhang in #469
• Disable whitesource checks as we migrated to synk by @sfc-gh-kdama in #473
• SNOW-645776 Run different versions of confluent and AK using matrix strategy + Refactor by @sfc-gh-japatel in #476

New Contributors

• @sfc-gh-zefan made their first contribution in #459
• @sfc-gh-kdama made their first contribution in #473

Full Changelog: v1.8.0...v1.8.1