1.1.4

What's Changed

• NO-SNOW Upgrade JDBC to 3.13.30 by @sfc-gh-japatel in #500

Full Changelog: v1.1.3...v1.1.4

v1.1.3

This is a patch release as we're seeing a regression in 1.1.2 that the Snowpipe Streaming channels will get invalidated a lot more often with low internet bandwidth, and the ingestion won't make progress unless customers increase internet bandwidth or reduce workload size.

What's Changed

• NO-SNOW Fix flaky Windows test by @sfc-gh-lsembera in #484
• Revert "SNOW-787565: Use dynamic scaling thread pool instead of fixed… by @sfc-gh-tzhang in #486
• Enable AZURE and GCP tests to run on windows machine by @sfc-gh-japatel in #483
• V1.1.3 release by @sfc-gh-tzhang in #487

Full Changelog: v1.1.2...v1.1.3

v1.1.2

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] Big refactor on the pom.xml, upgrade a few dependency versions, add compiling checker and cleanup some of the dependencies
• [Improvement] Improve tests to run on different cloud providers and different OSs
• [Improvement] Make the throttling threshold limit configurable
• [Improvement] Improve the insertRows performance
• [Improvement] Do not log customer column data in exception
• [Improvement] Use dynamic scaling thread pool instead of fixed thread pool to release resources
• [Improvement] Add support for jvm.nonProxy.hosts
• [Bug Fix] Remove a few Arrow key words since we're using Parquet by default now
• [Bug Fix] Fix an issue that causes file build failure due to encryption key changes
• [Bug Fix] Fix a bug in OnErrorOption.Abort to put data into the correct temporary buffer

What's Changed

• SNOW-765525 add compiling checker and clean up dependency by @sfc-gh-dwang in #359
• SNOW-759237 edit few log lines by @sfc-gh-gdoci in #364
• SNOW-766066 Run tests on Windows by @sfc-gh-lsembera in #369
• SNOW-744974: avoid combining channel data with different encryption key to the same chunk by @sfc-gh-tzhang in #337
• [Snyk] Security upgrade org.apache.hadoop:hadoop-common from 3.3.4 to 3.3.5 by @snyk-bot in #370
• Resolves bouncycastle related linkage errors by @sfc-gh-tjones in #424
• Build fixes by @sfc-gh-tjones in #428
• Fixes snyk issue identified in issue 422 by @sfc-gh-tjones in #430
• SNOW-773778 make the absolute size threshold limit for throttling configurable, default 200 MB by @sfc-gh-gdoci in #426
• Issue 421 Fixes Jettison vuln by @sfc-gh-tjones in #434
• Upgrades Jettison further by @sfc-gh-tjones in #437
• NO-SNOW: InsertRow perf improvements by @sfc-gh-tzhang in #450
• Add CLA bot by @sfc-gh-rcheng in #461
• @no-snow print stack trace for failing test (StreamingIngestChannelTe… by @sfc-gh-gdoci in #451
• SNOW-783754 Do not log column values in Exception + Pass in row Index in Error Message by @sfc-gh-japatel in #453
• SNOW-787565: Use dynamic scaling thread pool instead of fixed thread pool by @sfc-gh-tzhang in #470
• @no-snow fix bug in OnErrorOption.Abort for Parquet by @sfc-gh-gdoci in #452
• SNOW-726924 Re-add jvmNonProxy host by @sfc-gh-japatel in #474
• NO-SNOW: Preserve old JDBC behavior after upgrade by @sfc-gh-tzhang in #475
• @snow SNOW-779794 Client SDK Parquet buffer: fix perf degradation due to calling suboptimal lib method per row by @sfc-gh-azagrebin in #433
• [Snyk] Security upgrade net.snowflake:snowflake-jdbc from 3.13.25 to 3.13.29 by @snyk-bot in #476
• SNOW-766522 Use caffeine cache by @sfc-gh-lsembera in #367
• SNOW-765525 Add slf4j-simple back to test scope by @sfc-gh-lsembera in #397
• NO-SNOW Fix bug for ParquetRowBuffer regarding row Index by @sfc-gh-japatel in #477
• Adds back provided scope for slf4j-api by @sfc-gh-tjones in #431
• SNOW-745975 Run tests against Azure and GCP by @sfc-gh-japatel in #478
• V1.1.2 release by @sfc-gh-tzhang in #482

New Contributors

• @sfc-gh-tjones made their first contribution in #424

Full Changelog: v1.1.1...v1.1.2

v1.1.1

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix a nullCount/rowCount mismatch issue
• [Bug Fix] Fix an issue that causes Stream failure and Parquet scanner failure, need server side fix to be deployed as well
• [Improvement] Cache unquoted column names to improve insertRow performance
• [Improvement] Update some dependencies
• [Improvement] Some logging doc related improvement

What's Changed

• NO-SNOW: add logging to debug file uploading failure by @sfc-gh-tzhang in #340
• Clarify thread-safety guarantees in Javadoc by @sfc-gh-lsembera in #342
• Reduce info logging around JWT token regeneration by @sfc-gh-rcheng in #345
• SNOW-470176: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.14.0-rc1 to 2.14.0 by @sfc-gh-snyk-sca-sa in #347
• SNOW-761399: exclude nimbus-jose-jwt from hadoop-common dependency by @sfc-gh-dwang in #352
• @snow SNOW-759764 Streaming Ingest Parquet: use Snowflake column ordinal by @sfc-gh-azagrebin in #353
• SNOW-766522 Cache unquoted column names by @sfc-gh-lsembera in #362
• SNOW-735517 Fix nullCount/rowCount mismatch by @sfc-gh-lsembera in #357
• V1.1.1 release by @sfc-gh-tzhang in #365

New Contributors

• @sfc-gh-rcheng made their first contribution in #345
• @sfc-gh-dwang made their first contribution in #352

Full Changelog: v1.1.0...v1.1.1

v1.1.0

This is the first non-beta release after v0.10.8 and it contains the support to do Snowpipe Streaming

What's Changed

• Rename file after we used gh actions to pass deploy script by @sfc-gh-japatel in #93
• new semgrep integration by @sfc-gh-spandey in #95
• PRODSEC-1059 integrate whitesource for github by @sfc-gh-jfan in #97
• Remove guava dependency by @sfc-gh-japatel in #101
• PRODSEC-1192 add jira flow by @sfc-gh-jfan in #103
• SNOW-500221 Client Api SDK changes by @sfc-gh-anwang in #102
• Add interface for insertFiles with clientInfo - Along with tests by @sfc-gh-japatel in #105
• SNOW-488437 Add connection timeout by @sfc-gh-anwang in #104
• SNOW-514117: Add illegal argument exception for negative client sequencer by @sfc-gh-anwang in #106
• Release a beta version of SDK for new Snowpipe APIs by @sfc-gh-japatel in #107
• Snowpipe Streaming V1 by @sfc-gh-tzhang in #109
• Add nullability to the field metadata by @sfc-gh-mnaides in #111
• SNOW-356153 Move streaming ingest to use profile.json by @sfc-gh-mnaides in #112
• Validate JSON for Object and String Date parsing by @sfc-gh-mnaides in #113
• SNOW-533760: include client name and key during upload by @sfc-gh-tzhang in #114
• SNOW-534044 validate max object and variant sizes by @sfc-gh-mnaides in #115
• NO-SNOW: small fixes on configuration file behaviors by @sfc-gh-tzhang in #116
• SNOW-535803 Support TIMESTAMP_TZ in streaming ingest by @sfc-gh-mnaides in #122
• NO-SNOW: update dependency by @sfc-gh-tzhang in #123
• NO-SNOW: simplify SnowflakeStreamingIngestClient interface by @sfc-gh-tzhang in #117
• NO_SNOW: SDK beta release 0.10.5-beta with streaming support by @sfc-gh-tzhang in #125
• SNOW-518618: remove BC-FIPS dependency by @sfc-gh-tzhang in #124
• NO-SNOW: Support running unit tests when profile.json is missing by @sfc-gh-tzhang in #127
• SNOW-540567: Disable test to unblock beta release by @sfc-gh-tzhang in #128
• SNOW-540821 Fetch signed url on every put call for gcs by @sfc-gh-mnaides in #129
• Allow client to configure parameters by @sfc-gh-mnaides in #126
• SNOW-534105: Enable encryption by default by @sfc-gh-tzhang in #130
• Pass in row index corresponding to input rows in response by @sfc-gh-japatel in #131
• Release version 1.0.0-beta by @sfc-gh-japatel in #132
• SNOW-350701 Refresh credentials on correct error by @sfc-gh-mnaides in #133
• SNOW-515458 Fix min/max for binary datatype by @sfc-gh-mnaides in #134
• SNOW-552292 fix max string ep info when the string is truncated by @sfc-gh-mnaides in #135
• Increase the default throttling threshold by @sfc-gh-mnaides in #138
• NO-SNOW: multiple small adjustments/fixes for Snowpipe Streaming by @sfc-gh-tzhang in #136
• NO-SNOW: adding more logging + fix IT tests by @sfc-gh-tzhang in #139
• Add row sequencer for debugging by @sfc-gh-japatel in #141
• SNOW-556588 Make timestamp test more robust by @sfc-gh-mnaides in #142
• Debug submit flush task only when isNeedFlush is true to avoid loggin… by @sfc-gh-japatel in #137
• Release version 1.0.1-beta by @sfc-gh-tzhang in #143
• @snow SNOW-554959 Align chunk encryption with block size and count for the whole interleaved file by @sfc-gh-azagrebin in #140
• SNOW-557106 enable jmx metrics for streaming by @sfc-gh-japatel in #144
• NO-SNOW: update code owner by @sfc-gh-tzhang in #147
• SNOW-560665: remove usage of instanceof for type checking by @sfc-gh-tzhang in #148
• SNOW-564031: Incorrect EP for NULL columns causing BDEC registration failure by @sfc-gh-tzhang in #149
• SNOW-470176 Update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.2.1 by @mend-for-github-com in #151
• nosnow- support string date formats by @sfc-gh-mnaides in #153
• SNOW-566797: Support SDK with proxy server by @sfc-gh-tzhang in #154
• Release version 1.0.2-beta by @sfc-gh-tzhang in #150
• Enable ignored tests which run against sfctest0 prod1 account by @sfc-gh-japatel in #108
• no-snow fix error message printing by @sfc-gh-japatel in #157
• @snow SNOW-572229 Streaning Ingest: Manual padding of compressed chunk for encryption by @sfc-gh-azagrebin in #159
• NO-SNOW: add retry logic on 429 (TOO_MANY_REQUESTS) by @sfc-gh-tzhang in #160
• NO-SNOW: fix regression after the JMX metrics change by @sfc-gh-tzhang in #161
• NO-SNOW: Format code by @sfc-gh-kbregula in #155
• SNOW-556750 retry certain streaming ingest specific retries by @sfc-gh-mnaides in #158
• Use CloseableHttpClient and consume Entity on all APIs by @sfc-gh-japatel in #167
• NO_SNOW: fix bug in channel.close by @sfc-gh-tzhang in #163
• NO-SNOW: increase retry count due to server side Commit Service not making progress by @sfc-gh-tzhang in #172
• Fix logging project properties load exception by @sfc-gh-azagrebin in #171
• Release version 1.0.2-beta.1 by @sfc-gh-tzhang in #169
• NO-SNOW: accept account names with hyphen by @sfc-gh-tzhang in #174
• NO-SNOW: A few small fixes on Snowpipe Streaming PrPr by @sfc-gh-tzhang in #175
• no-snow Add readme and call offsetToken API in Example file by @sfc-gh-japatel in #177
• NO-SNOW: Update Arrow library to 7.0.0 to support VectorSchemaRootAppender by @sfc-gh-tzhang in #180
• SNOW-592855, SNOW-606515: fix channel.close time out issue due to high upload latency by @sfc-gh-tzhang in #182
• NO-SNOW: fix test failure due to time zone difference by @sfc-gh-tzhang in #183
• NO-SNOW: debug memory issue by @sfc-gh-tzhang in #185
• PRODSEC-1251 Whitesource Transition to Snyk by @sfc-gh-jfan in #170
• SNOW-610062: use "/" as name separator instead of using Paths.get by @sfc-gh-tzhang in #187
• SNOW-608483 fix column stats for collated strings by @sfc-gh-mnaides in #186
• NO-SNOW: improve README and Snowpipe Streaming Example by @sfc-gh-tzhang in #189
• V1.0.2-beta.2 release by @sfc-gh-tzhang in #190
• prodsec-1251 minor update on jira token by @sfc-gh-jfan in https://...

v1.0.3-beta.1

The 1.0.3-beta release has an issue with GCS on JDBC 3.13.18, so we revert #334 and then create a new release on top

What's Changed

• SNOW-663704 Send hex-encoded min/max for BINARY by @sfc-gh-lsembera in #329
• Remove JAXB by @sfc-gh-lsembera in #332
• NO-SNOW: revert https://github.com/snowflakedb/snowflake-kafka-connector/pull/546/files as it's causing failure on GCS by @sfc-gh-tzhang in #334
• V1.0.3-beta.1 release by @sfc-gh-tzhang in #335

Full Changelog: v1.0.3-beta...v1.0.3-beta.1

v1.0.3-beta

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] In order to increase timestamp parsing performance, the number of accepted timestamp formats has been limited. Please see the Javadoc for SnowflakeStreamingIngestChannel#insertRows for more details.
• [Improvement] Add column name to data validation error messages
• [Improvement] Convert empty variant strings into variant column to NULL
• [Improvement] Avoid datetime parsing wherever possible to improve performance
• [Improvement] Allow inf, -inf and nan strings into FLOAT columns
• [Improvement] Add telemetry to track the end2end latency
• [Fixes] Fix small parquet binary max length inconsistency with Arrow
• [Improvement] Add option to use parquet file writer per channel and then merge them into one per chunk
• [Improvement] Improve build latency for high throughput case with many channels
• [Fix] Fix various issues with unicode strings
• [Fix] Ingest SDK Does Not Honor http.nonProxyHosts JVM Argument
• [Improvement] Enable Parquet file by default and enforce single rowcount Parquet files

This release contains a few bug fixes and improvements for Snowpipe:

• [Fix] SimpleIngestIT runtime created database

What's Changed

• SNOW-710291 Allow inf, -inf and nan strings into FLOAT columns by @sfc-gh-lsembera in #306
• SNOW-684267 Add column name to data validation error messages by @sfc-gh-asen in #309
• SNOW-711338 Convert empty variant strings into Variant column to NULL by @sfc-gh-asen in #310
• SNOW-710401 Avoid datetime parsing wherever possible by @sfc-gh-lsembera in #311
• SNOW-701710: [Client side] Add telemetry to track the end2end latency for every BDEC by @sfc-gh-tzhang in #295
• SNOW-690793 use parquet file writer per channel and then merge them i… by @sfc-gh-gdoci in #299
• @no-snow fix small parquet binary max length inconsistency with Arrow by @sfc-gh-gdoci in #319
• SNOW-715618: Improve build latency for high throughput case with many channels by @sfc-gh-tzhang in #321
• SNOW-663621 Fix problems with unicode strings by @sfc-gh-lsembera in #300
• Enable Parquet data type tests by @sfc-gh-lsembera in #313
• [SNOW-726993] Ingest SDK Does Not Honor http.nonProxyHosts JVM Argument by @sfc-gh-wfateem in #325
• SNOW-731571 SimpleIngestIT runtime created database by @sfc-gh-lozhang in #327
• SNOW-620624 enforce single rowcount Parquet files by @sfc-gh-gdoci in #324
• SNOW-733699 make Parquet the default blob format version instead of Arrow by @sfc-gh-gdoci in #322
• SNOW-711299 New date/time processing by @sfc-gh-lsembera in #320
• NO-SNOW: various fixes/improvements in the SDK by @sfc-gh-tzhang in #330
• V1.0.3-beta release by @sfc-gh-tzhang in #326

New Contributors

• @sfc-gh-asen made their first contribution in #309
• @sfc-gh-wfateem made their first contribution in #325
• @sfc-gh-lozhang made their first contribution in #327

Full Changelog: v1.0.2-beta.7...v1.0.3-beta

v1.0.2-beta.7

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix incorrect behaviors for table with quoted columns
• [Bug Fix] Fix binary data type for Parquet
• [Bug Fix] Fix incorrect behaviors when sending telemetry data to Snowflake
• [Bug Fix] Fix incorrect behaviors for TIMESTAMP fractional part
• [Improvement] Remove circular dependency on channel object
• [Improvement] Add filename in Parquet MD to support

What's Changed

• SNOW-637237 Test opening several thousand channels in parallel by @sfc-gh-lsembera in #294
• @snow SNOW-657667 Streaming ingest: factor out channel from the buffer by @sfc-gh-azagrebin in #285
• SNOW-696155 fix binary values for Parquet by @sfc-gh-gdoci in #292
• SNOW-647377 Support quoted column names by @sfc-gh-lsembera in #293
• SNOW-703928 add two system properties for IT tests by @sfc-gh-gdoci in #296
• @snow SNOW-705508 Exclude slf4j-reload4j from Hadoop dep by @sfc-gh-azagrebin in #301
• NO-SNOW: fix client telemetry logic by @sfc-gh-tzhang in #297
• SNOW-706682 Log retry-related messages as INFO by @sfc-gh-lsembera in #302
• SNOW-705594: fix ConcurrentModificationException on the telemetry hashmap by @sfc-gh-tzhang in #303
• V1.0.2-beta.7 release by @sfc-gh-tzhang in #305
• SNOW-707520 Fix incorrect fractional part calculation of TIMESTAMP by @sfc-gh-lsembera in #304
• SNOW-704566 Add filename in Parquet extra MD by @sfc-gh-kkloudas in #298

Full Changelog: v1.0.2-beta.6...v1.0.2-beta.7

v1.0.2-beta.6

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] Add parquet file support, this will be our default file format in the future
• [Improvement] Add support to specify a memory limit per client instead of using the system max memory
• [Improvement] Remove a bunch of unused code and setup code coverage

This release contains a few bug fixes and improvements for both Snowpipe and Snowpipe Streaming:

• [Improvement] Improve the HTTP retry logic

What's Changed

• SNOW-470176: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.13.2.1 to 2.13.4 by @sfc-gh-snyk-sca-sa in #237
• @snow SNOW-633432 Parquet file generator by @sfc-gh-azagrebin in #200
• SNOW-667402:Semgrep workflow update by @sfc-gh-hpathak in #234
• NO-SNOW: add support to reset a rowIndex in the error response whenever needed by @sfc-gh-tzhang in #241
• Revert "@snow SNOW-633432 Parquet file generator (#200)" by @sfc-gh-gdoci in #249
• Introduce parquet file generator by @sfc-gh-gdoci in #251
• Revert "Introduce parquet file generator (#251)" by @sfc-gh-tzhang in #269
• [Snyk] Security upgrade org.apache.arrow:arrow-compression from 9.0.0 to 10.0.0 by @snyk-bot in #271
• SNOW-637237 Thread-safe HTTP client initialisation by @sfc-gh-lsembera in #277
• SNOW-684474: Support adding a memory limit through the config file by @sfc-gh-tzhang in #279
• @snow SNOW-633432 Parquet file generator (fixed dep) by @sfc-gh-azagrebin in #280
• SNOW-637237 Retry for 2 minutes since the first failure by @sfc-gh-lsembera in #281
• Revert "@snow SNOW-612516 Streaming ingest: add random file reader to scan single columns, format version 2 by @sfc-gh-azagrebin in #282
• NO-SNOW: enable code coverage check in the SDK by @sfc-gh-tzhang in #283
• @snow no-snow Streaming ingest: always send BDEC version as server side supports this by @sfc-gh-azagrebin in #284
• @snow no-snow Fix more parquet dependencies causing vulnerabilities by @sfc-gh-azagrebin in #287
• V1.0.2-beta.6 release by @sfc-gh-tzhang in #286

New Contributors

• @sfc-gh-hpathak made their first contribution in #234
• @snyk-bot made their first contribution in #271

Full Changelog: v1.0.2-beta.5...v1.0.2-beta.6

v1.0.2-beta.5

Please Note: This is a release on top of v1.0.2-beta.4
This release contains a few bug fixes and improvements for Snowpipe Streaming

[Improvement] Improve and fix a few data type validation logic
[Improvement] Add throttling logic on direct memory based on uploading queue size
[Improvement] Add basic logic to support Parquet file format, this work is still in progress and we're still using Arrow
[Improvement] Improve exception handling logic

What's Changed

• PRODSEC-1924 rm whitesource legacy files by @sfc-gh-jfan in #217
• @snow SNOW-657206 SteamingIngestIT: add random suffix to DB name to fix concurrent CI runs by @sfc-gh-azagrebin in #222
• @snow SNOW-633432 Refactor RowBuffer and Flusher API by @sfc-gh-azagrebin in #202
• @snow SNOW-633432 Refactor tests to support various BDEC format versions by @sfc-gh-azagrebin in #209
• SNOW-470176: [Snyk] Fix for 2 vulnerabilities by @sfc-gh-snyk-sca-sa in #225
• SNOW-655614 Data type validation: Boolean by @sfc-gh-lsembera in #228
• SNOW-655614 Data type validation: Date and Time by @sfc-gh-lsembera in #229
• SNOW-655614 Data type validation: NUMBER and REAL by @sfc-gh-lsembera in #230
• SNOW-655614 Data type validation: String and Binary by @sfc-gh-lsembera in #231
• SNOW-616916: throttle the insertRows API when the flush task queue size reaches a certain threshold by @sfc-gh-tzhang in #215
• NO-SNOW: add stacktrace during blob building failure by @sfc-gh-tzhang in #232
• SNOW-655614 Data type validation: Semi-structured types by @sfc-gh-lsembera in #233
• V1.0.2-beta.5 release by @sfc-gh-tzhang in #235

New Contributors

• @sfc-gh-snyk-sca-sa made their first contribution in #225
• @sfc-gh-lsembera made their first contribution in #228

Full Changelog: v1.0.2-beta.4...v1.0.2-beta.5