v2.0.4

This release contains a few improvements and bug fixes for Snowpipe Streaming:

• [Improvement] Support a new ON_ERROR option SKIP_BATCH, which will skip the entire batch if there is any issue and return all errors as part of the response
• [Improvement] Add row index information to all exceptions
• [Improvement] Upgrade snappy-java dependency
• [Improvement] Add a new interface to return the table schema information for a channel
• [Improvement] Add a new configuration option MAX_CLIENT_LAG which can contain the flush frequency, by default we flush every second
• [Bug Fix] Fix an issue when using snowflake-jdbc-fips
• [Bug Fix] Fix a rare ConcurrentModificationException issue
• [Bug Fix] Fix two issues in insertRows API that might cause wrong results in a very rare case
• [Bug Fix] Limit the max allowed number of chunks in blob to avoid the case when the request is too large

What's Changed

• Remove Public Preview note by @sfc-gh-lsembera in #582
• @no-snow refactor a few variables into ClientBufferParameters by @sfc-gh-gdoci in #581
• SNOW-914666 Adds MAX_CLIENT_LAG configuration option by @sfc-gh-tjones in #586
• SNOW-919423 Surfaces Table Schema for a Channel by @sfc-gh-tjones in #589
• NO-SNOW Remove wrong documentation by @sfc-gh-lthiede in #591
• updated SDK version to 2.0.4-SNAPSHOT by @sfc-gh-dshah in #588
• SNOW-924864 Upgrade snappy-java by @sfc-gh-lsembera in #592
• SNOW-918949: Add row index information to all exceptions by @sfc-gh-tzhang in #590
• SNOW-928802 Use concurrent hashmap to prevent ConcurrentModificationE… by @sfc-gh-japatel in #594
• SNOW-926149 Fix issues while using snowflake-jdbc-fips by @sfc-gh-lsembera in #596
• Update readme for JDBC version issues #599 by @sfc-gh-xhuang in #600
• SNOW-906224: Add new ON_ERROR=SKIP_BATCH option by @sfc-gh-tzhang in #597
• NO-SNOW: Fix two issues in insertRows API by @sfc-gh-tzhang in #602
• SNOW-902709 Limit the max allowed number of chunks in blob by @sfc-gh-lsembera in #580
• PRODSEC-3611 fix GHA parsing by @sfc-gh-jfan in #603
• no-snow Sort pom.xml in format.sh by @sfc-gh-lsembera in #604
• SNOW-936900 Use upgraded version of snappy-java by @sfc-gh-lsembera in #605
• SNOW-945927 SNOW-945928 Fix Snyk vulnerabilities by @sfc-gh-lsembera in #641
• NO-SNOW: fix row index issue for ON_ERROR=SKIP_BATCH by @sfc-gh-tzhang in #606
• V2.0.4 release by @sfc-gh-tzhang in #643

New Contributors

• @sfc-gh-lthiede made their first contribution in #591

Full Changelog: v2.0.3...v2.0.4

v2.0.3

This release contains a few improvements and bug fixes for Snowpipe Streaming:

• [Improvement] Support OAuth Authentication
• [Improvement] Remove exactly once related code for Snowpipe
• [Improvement] Support publishing unshaded snapshot release to Nexus repo
• [Improvement] Add retry logic for invalid JWT token
• [Improvement] Add warning for large batches in insertRows
• [Bug Fix] Fix a NPE issue caused by race condition

What's Changed

• SNOW-352846 OAuth Authentication: #1 Polymorph auth credential by @sfc-gh-alhuang in #532
• SNOW-693813 disable client info snowpipe tests, remove later by @sfc-gh-japatel in #536
• [SNOW-693813] Remove exactly once methods calling into server by @sfc-gh-rcheng in #539
• SNOW-855191 Race condition causing NPE by @sfc-gh-lsembera in #544
• SNOW-352846 OAuth Authentication: #2 OAuth Support by @sfc-gh-alhuang in #537
• SNOW-870386 Add default scheme parameters for test utils when profile is not specified by @sfc-gh-alhuang in #552
• SNOW-352846 OAuth Authentication: #3 Add error message by @sfc-gh-alhuang in #553
• NO-SNOW: Support publishing unshaded snapshot release to Nexus repo by @sfc-gh-tzhang in #561
• Update AWS test profile by @sfc-gh-alhuang in #564
• changed version from 2.0.2 to 2.0.3-SNAPSHOT by @sfc-gh-dshah in #565
• SNOW-877039 Ingest SDK retry for INVALID JWT Response by @sfc-gh-alhuang in #562
• Move testJWTRetries test to IT by @sfc-gh-alhuang in #572
• SNOW-861190 Add warning for large batches in insertRows by @sfc-gh-lsembera in #551
• Using jenkins sonatype username and password instead of my own by @sfc-gh-dshah in #571
• NO-SNOW: Add a few requested changes by @sfc-gh-tzhang in #576
• V2.0.3 release by @sfc-gh-tzhang in #577

New Contributors

• @sfc-gh-alhuang made their first contribution in #532

Full Changelog: v2.0.2...v2.0.3

v2.0.2

This release contains a few improvements for Snowpipe Streaming:

• [Improvement] Update dependencies based on Wiz and Snyk vulnerability scan result
• [Improvement] Improve retry logic on exceptions like SSLException
• [Improvement] Make the role as an optional input and support using of default role associated with the user
• [Improvement] Send uncompressed chunk length to server side for tracking purpose

What's Changed

• Remove CLA bot and update README.md to include cla and formatter requirements by @sfc-gh-rcheng in #510
• SNOW-841883 Upgrade snappy-java to 1.1.10.1 by @sfc-gh-lsembera in #533
• [Snyk] Security upgrade com.google.guava:guava from 31.1-jre to 32.0.0-jre by @sfc-gh-snyk-sca-sa in #534
• NO-SNOW Bump version to 2.0.2-SNAPSHOT by @sfc-gh-lsembera in #529
• Use later version of google guava, as recommended by CVE by @sfc-gh-japatel in #535
• [SNOW-843760] Upgrade netty-common in response to CVE vulnerability by @sfc-gh-rcheng in #538
• NO-SNOW Add note about dependencies by @sfc-gh-lsembera in #540
• SNOW-851351 Retry javax.net.ssl.SSLException by @sfc-gh-lsembera in #543
• [SNOW-843760] Update pom.xml corresponding to Wiz vulnerability scan by @sfc-gh-rcheng in #546
• SNOW-532834 Using default role when snowflake.role.name is not specified by @sfc-gh-alhuang in #507
• NO-SNOW Retry failed requests more aggressively by @sfc-gh-lsembera in #550
• v2.0.2 version upgrade by @sfc-gh-tzhang in #557
• @snow SNOW-835618 Snowpipe Streaming: send uncompressed chunk length from SDK to GS by @sfc-gh-azagrebin in #521
• reduce number of rows in BigFilesIT by @sfc-gh-gdoci in #559

Full Changelog: v2.0.1...v2.0.2

v2.0.1

This is a patch release on top of the v2.0.0 where we fix an unexpected dependency behavior for Snowflake JDBC

What's Changed

• @snow SNOW-837436 Streaming ingest: dedup var for support status code messages by @sfc-gh-azagrebin in #526
• NO-SNOW Specify dependencies in public_pom.xml by @sfc-gh-lsembera in #527
• V2.0.1 release by @sfc-gh-tzhang in #528

Full Changelog: v2.0.0...v2.0.1

v2.0.0

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] Improving the dependencies shading and relocating logic, it introduces some behavior changes (e.g. JDBC) and that's why we bump the major version
• [Improvement] Make a few channel/chunk/file size limit parameters configurable
• [Improvement] Add more telemetries to track end2end latency
• [Improvement] Support GCS downscoped token
• [Improvement] Clean up all Arrow related code
• [Improvement] Add Attribution notice
• [Improvement] Enforce allowed DATE and TIMESTAMP range
• [Improvement] Expose more server side channel invalidation error message for customers to self-mitigate
• [Bug Fix] Fix an issue where some BG threads are not stopped during exception

What's Changed

• Adding a script to release unshaded version into S3 repo by @sfc-gh-hqin in #346
• Update protobuf version to 3.19.6 by @sfc-gh-dwang in #488
• SNOW-807102 Disable OpenManyChannelsIT by @sfc-gh-lsembera in #491
• SNOW-801398 Tidy up shaded jar by @sfc-gh-lsembera in #490
• SNOW-806047 fix logged uncompressedSize for Parquet by @sfc-gh-gdoci in #489
• SNOW-811371 Shade and Relocate Snowflake JDBC Classes by @sfc-gh-wfateem in #494
• NO-SNOW: Update MAX_CHUNK_SIZE_IN_BYTES default to be 32MB and make it configurable by @sfc-gh-tzhang in #498
• SNOW-758492: avoid combining channels with different schema by @sfc-gh-tzhang in #501
• Dshah snow 801462 snapshotdeploy by @sfc-gh-dshah in #496
• Revert "SNOW-811371 Shade and Relocate Snowflake JDBC Classes (#494)" by @sfc-gh-lsembera in #503
• Fix 'Connection reset' build errors by caching mvn dependencies by @sfc-gh-lsembera in #502
• [SNOW-766867] Add blob latency and start timestamps to registerBlobs call by @sfc-gh-rcheng in #429
• SNOW-768197 Attribution notice by @sfc-gh-lsembera in #506
• @snow SNOW-763199 Enable StreamingIngestIT.testTableColumnEvolution by @sfc-gh-azagrebin in #505
• NO-SNOW minor error text typo fix by @sfc-gh-xhuang in #508
• SNOW-768195: [Client Side] Support GCS downscoped token for Snowpipe Streaming by @sfc-gh-tzhang in #504
• SNOW-818891 Remove Arrow and other unneeded dependencies by @sfc-gh-lsembera in #497
• SNOW-747848: Cleanup BG threads with exceptions by @sfc-gh-tzhang in #479
• SNOW-830955 Use default SDK version in User-Agent by @sfc-gh-lsembera in #513
• NO-SNOW Exclude examples from code coverage scanning by @sfc-gh-lsembera in #515
• SNOW-830862 Maven build improvements by @sfc-gh-lsembera in #514
• SNOW-768197 Attribution notice for shaded dependencies by @sfc-gh-lsembera in #517
• NO-SNOW: Limit the chunk size to avoid potential OOM issue in the Parquet Reader by @sfc-gh-tzhang in #516
• NO-SNOW Enforce allowed DATE and TIMESTAMP range by @sfc-gh-lsembera in #520
• SNOW-720420 Enforce per-row size limits by @sfc-gh-lsembera in #519
• NO-SNOW Mention minimal JDBC driver version 3.13.30 by @sfc-gh-lsembera in #523
• V2.0.0 release by @sfc-gh-tzhang in #522
• NO-SNOW Exclude additional example from code coverage by @sfc-gh-xhuang in #524
• @snow SNOW-837436 Streaming ingest: get status code message in channel status response by @sfc-gh-azagrebin in #525

New Contributors

• @sfc-gh-hqin made their first contribution in #346
• @sfc-gh-dshah made their first contribution in #496
• @sfc-gh-xhuang made their first contribution in #508

Full Changelog: v1.1.3...v2.0.0

1.1.4

What's Changed

• NO-SNOW Upgrade JDBC to 3.13.30 by @sfc-gh-japatel in #500

Full Changelog: v1.1.3...v1.1.4

v1.1.3

This is a patch release as we're seeing a regression in 1.1.2 that the Snowpipe Streaming channels will get invalidated a lot more often with low internet bandwidth, and the ingestion won't make progress unless customers increase internet bandwidth or reduce workload size.

What's Changed

• NO-SNOW Fix flaky Windows test by @sfc-gh-lsembera in #484
• Revert "SNOW-787565: Use dynamic scaling thread pool instead of fixed… by @sfc-gh-tzhang in #486
• Enable AZURE and GCP tests to run on windows machine by @sfc-gh-japatel in #483
• V1.1.3 release by @sfc-gh-tzhang in #487

Full Changelog: v1.1.2...v1.1.3

v1.1.2

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] Big refactor on the pom.xml, upgrade a few dependency versions, add compiling checker and cleanup some of the dependencies
• [Improvement] Improve tests to run on different cloud providers and different OSs
• [Improvement] Make the throttling threshold limit configurable
• [Improvement] Improve the insertRows performance
• [Improvement] Do not log customer column data in exception
• [Improvement] Use dynamic scaling thread pool instead of fixed thread pool to release resources
• [Improvement] Add support for jvm.nonProxy.hosts
• [Bug Fix] Remove a few Arrow key words since we're using Parquet by default now
• [Bug Fix] Fix an issue that causes file build failure due to encryption key changes
• [Bug Fix] Fix a bug in OnErrorOption.Abort to put data into the correct temporary buffer

What's Changed

• SNOW-765525 add compiling checker and clean up dependency by @sfc-gh-dwang in #359
• SNOW-759237 edit few log lines by @sfc-gh-gdoci in #364
• SNOW-766066 Run tests on Windows by @sfc-gh-lsembera in #369
• SNOW-744974: avoid combining channel data with different encryption key to the same chunk by @sfc-gh-tzhang in #337
• [Snyk] Security upgrade org.apache.hadoop:hadoop-common from 3.3.4 to 3.3.5 by @snyk-bot in #370
• Resolves bouncycastle related linkage errors by @sfc-gh-tjones in #424
• Build fixes by @sfc-gh-tjones in #428
• Fixes snyk issue identified in issue 422 by @sfc-gh-tjones in #430
• SNOW-773778 make the absolute size threshold limit for throttling configurable, default 200 MB by @sfc-gh-gdoci in #426
• Issue 421 Fixes Jettison vuln by @sfc-gh-tjones in #434
• Upgrades Jettison further by @sfc-gh-tjones in #437
• NO-SNOW: InsertRow perf improvements by @sfc-gh-tzhang in #450
• Add CLA bot by @sfc-gh-rcheng in #461
• @no-snow print stack trace for failing test (StreamingIngestChannelTe… by @sfc-gh-gdoci in #451
• SNOW-783754 Do not log column values in Exception + Pass in row Index in Error Message by @sfc-gh-japatel in #453
• SNOW-787565: Use dynamic scaling thread pool instead of fixed thread pool by @sfc-gh-tzhang in #470
• @no-snow fix bug in OnErrorOption.Abort for Parquet by @sfc-gh-gdoci in #452
• SNOW-726924 Re-add jvmNonProxy host by @sfc-gh-japatel in #474
• NO-SNOW: Preserve old JDBC behavior after upgrade by @sfc-gh-tzhang in #475
• @snow SNOW-779794 Client SDK Parquet buffer: fix perf degradation due to calling suboptimal lib method per row by @sfc-gh-azagrebin in #433
• [Snyk] Security upgrade net.snowflake:snowflake-jdbc from 3.13.25 to 3.13.29 by @snyk-bot in #476
• SNOW-766522 Use caffeine cache by @sfc-gh-lsembera in #367
• SNOW-765525 Add slf4j-simple back to test scope by @sfc-gh-lsembera in #397
• NO-SNOW Fix bug for ParquetRowBuffer regarding row Index by @sfc-gh-japatel in #477
• Adds back provided scope for slf4j-api by @sfc-gh-tjones in #431
• SNOW-745975 Run tests against Azure and GCP by @sfc-gh-japatel in #478
• V1.1.2 release by @sfc-gh-tzhang in #482

New Contributors

• @sfc-gh-tjones made their first contribution in #424

Full Changelog: v1.1.1...v1.1.2

v1.1.1

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix a nullCount/rowCount mismatch issue
• [Bug Fix] Fix an issue that causes Stream failure and Parquet scanner failure, need server side fix to be deployed as well
• [Improvement] Cache unquoted column names to improve insertRow performance
• [Improvement] Update some dependencies
• [Improvement] Some logging doc related improvement

What's Changed

• NO-SNOW: add logging to debug file uploading failure by @sfc-gh-tzhang in #340
• Clarify thread-safety guarantees in Javadoc by @sfc-gh-lsembera in #342
• Reduce info logging around JWT token regeneration by @sfc-gh-rcheng in #345
• SNOW-470176: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.14.0-rc1 to 2.14.0 by @sfc-gh-snyk-sca-sa in #347
• SNOW-761399: exclude nimbus-jose-jwt from hadoop-common dependency by @sfc-gh-dwang in #352
• @snow SNOW-759764 Streaming Ingest Parquet: use Snowflake column ordinal by @sfc-gh-azagrebin in #353
• SNOW-766522 Cache unquoted column names by @sfc-gh-lsembera in #362
• SNOW-735517 Fix nullCount/rowCount mismatch by @sfc-gh-lsembera in #357
• V1.1.1 release by @sfc-gh-tzhang in #365

New Contributors

• @sfc-gh-rcheng made their first contribution in #345
• @sfc-gh-dwang made their first contribution in #352

Full Changelog: v1.1.0...v1.1.1

v1.1.0

This is the first non-beta release after v0.10.8 and it contains the support to do Snowpipe Streaming

What's Changed

• Rename file after we used gh actions to pass deploy script by @sfc-gh-japatel in #93
• new semgrep integration by @sfc-gh-spandey in #95
• PRODSEC-1059 integrate whitesource for github by @sfc-gh-jfan in #97
• Remove guava dependency by @sfc-gh-japatel in #101
• PRODSEC-1192 add jira flow by @sfc-gh-jfan in #103
• SNOW-500221 Client Api SDK changes by @sfc-gh-anwang in #102
• Add interface for insertFiles with clientInfo - Along with tests by @sfc-gh-japatel in #105
• SNOW-488437 Add connection timeout by @sfc-gh-anwang in #104
• SNOW-514117: Add illegal argument exception for negative client sequencer by @sfc-gh-anwang in #106
• Release a beta version of SDK for new Snowpipe APIs by @sfc-gh-japatel in #107
• Snowpipe Streaming V1 by @sfc-gh-tzhang in #109
• Add nullability to the field metadata by @sfc-gh-mnaides in #111
• SNOW-356153 Move streaming ingest to use profile.json by @sfc-gh-mnaides in #112
• Validate JSON for Object and String Date parsing by @sfc-gh-mnaides in #113
• SNOW-533760: include client name and key during upload by @sfc-gh-tzhang in #114
• SNOW-534044 validate max object and variant sizes by @sfc-gh-mnaides in #115
• NO-SNOW: small fixes on configuration file behaviors by @sfc-gh-tzhang in #116
• SNOW-535803 Support TIMESTAMP_TZ in streaming ingest by @sfc-gh-mnaides in #122
• NO-SNOW: update dependency by @sfc-gh-tzhang in #123
• NO-SNOW: simplify SnowflakeStreamingIngestClient interface by @sfc-gh-tzhang in #117
• NO_SNOW: SDK beta release 0.10.5-beta with streaming support by @sfc-gh-tzhang in #125
• SNOW-518618: remove BC-FIPS dependency by @sfc-gh-tzhang in #124
• NO-SNOW: Support running unit tests when profile.json is missing by @sfc-gh-tzhang in #127
• SNOW-540567: Disable test to unblock beta release by @sfc-gh-tzhang in #128
• SNOW-540821 Fetch signed url on every put call for gcs by @sfc-gh-mnaides in #129
• Allow client to configure parameters by @sfc-gh-mnaides in #126
• SNOW-534105: Enable encryption by default by @sfc-gh-tzhang in #130
• Pass in row index corresponding to input rows in response by @sfc-gh-japatel in #131
• Release version 1.0.0-beta by @sfc-gh-japatel in #132
• SNOW-350701 Refresh credentials on correct error by @sfc-gh-mnaides in #133
• SNOW-515458 Fix min/max for binary datatype by @sfc-gh-mnaides in #134
• SNOW-552292 fix max string ep info when the string is truncated by @sfc-gh-mnaides in #135
• Increase the default throttling threshold by @sfc-gh-mnaides in #138
• NO-SNOW: multiple small adjustments/fixes for Snowpipe Streaming by @sfc-gh-tzhang in #136
• NO-SNOW: adding more logging + fix IT tests by @sfc-gh-tzhang in #139
• Add row sequencer for debugging by @sfc-gh-japatel in #141
• SNOW-556588 Make timestamp test more robust by @sfc-gh-mnaides in #142
• Debug submit flush task only when isNeedFlush is true to avoid loggin… by @sfc-gh-japatel in #137
• Release version 1.0.1-beta by @sfc-gh-tzhang in #143
• @snow SNOW-554959 Align chunk encryption with block size and count for the whole interleaved file by @sfc-gh-azagrebin in #140
• SNOW-557106 enable jmx metrics for streaming by @sfc-gh-japatel in #144
• NO-SNOW: update code owner by @sfc-gh-tzhang in #147
• SNOW-560665: remove usage of instanceof for type checking by @sfc-gh-tzhang in #148
• SNOW-564031: Incorrect EP for NULL columns causing BDEC registration failure by @sfc-gh-tzhang in #149
• SNOW-470176 Update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.2.1 by @mend-for-github-com in #151
• nosnow- support string date formats by @sfc-gh-mnaides in #153
• SNOW-566797: Support SDK with proxy server by @sfc-gh-tzhang in #154
• Release version 1.0.2-beta by @sfc-gh-tzhang in #150
• Enable ignored tests which run against sfctest0 prod1 account by @sfc-gh-japatel in #108
• no-snow fix error message printing by @sfc-gh-japatel in #157
• @snow SNOW-572229 Streaning Ingest: Manual padding of compressed chunk for encryption by @sfc-gh-azagrebin in #159
• NO-SNOW: add retry logic on 429 (TOO_MANY_REQUESTS) by @sfc-gh-tzhang in #160
• NO-SNOW: fix regression after the JMX metrics change by @sfc-gh-tzhang in #161
• NO-SNOW: Format code by @sfc-gh-kbregula in #155
• SNOW-556750 retry certain streaming ingest specific retries by @sfc-gh-mnaides in #158
• Use CloseableHttpClient and consume Entity on all APIs by @sfc-gh-japatel in #167
• NO_SNOW: fix bug in channel.close by @sfc-gh-tzhang in #163
• NO-SNOW: increase retry count due to server side Commit Service not making progress by @sfc-gh-tzhang in #172
• Fix logging project properties load exception by @sfc-gh-azagrebin in #171
• Release version 1.0.2-beta.1 by @sfc-gh-tzhang in #169
• NO-SNOW: accept account names with hyphen by @sfc-gh-tzhang in #174
• NO-SNOW: A few small fixes on Snowpipe Streaming PrPr by @sfc-gh-tzhang in #175
• no-snow Add readme and call offsetToken API in Example file by @sfc-gh-japatel in #177
• NO-SNOW: Update Arrow library to 7.0.0 to support VectorSchemaRootAppender by @sfc-gh-tzhang in #180
• SNOW-592855, SNOW-606515: fix channel.close time out issue due to high upload latency by @sfc-gh-tzhang in #182
• NO-SNOW: fix test failure due to time zone difference by @sfc-gh-tzhang in #183
• NO-SNOW: debug memory issue by @sfc-gh-tzhang in #185
• PRODSEC-1251 Whitesource Transition to Snyk by @sfc-gh-jfan in #170
• SNOW-610062: use "/" as name separator instead of using Paths.get by @sfc-gh-tzhang in #187
• SNOW-608483 fix column stats for collated strings by @sfc-gh-mnaides in #186
• NO-SNOW: improve README and Snowpipe Streaming Example by @sfc-gh-tzhang in #189
• V1.0.2-beta.2 release by @sfc-gh-tzhang in #190
• prodsec-1251 minor update on jira token by @sfc-gh-jfan in https://...