v1.1.1

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix a nullCount/rowCount mismatch issue
• [Bug Fix] Fix an issue that causes Stream failure and Parquet scanner failure, need server side fix to be deployed as well
• [Improvement] Cache unquoted column names to improve insertRow performance
• [Improvement] Update some dependencies
• [Improvement] Some logging doc related improvement

What's Changed

• NO-SNOW: add logging to debug file uploading failure by @sfc-gh-tzhang in #340
• Clarify thread-safety guarantees in Javadoc by @sfc-gh-lsembera in #342
• Reduce info logging around JWT token regeneration by @sfc-gh-rcheng in #345
• SNOW-470176: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.14.0-rc1 to 2.14.0 by @sfc-gh-snyk-sca-sa in #347
• SNOW-761399: exclude nimbus-jose-jwt from hadoop-common dependency by @sfc-gh-dwang in #352
• @snow SNOW-759764 Streaming Ingest Parquet: use Snowflake column ordinal by @sfc-gh-azagrebin in #353
• SNOW-766522 Cache unquoted column names by @sfc-gh-lsembera in #362
• SNOW-735517 Fix nullCount/rowCount mismatch by @sfc-gh-lsembera in #357
• V1.1.1 release by @sfc-gh-tzhang in #365

New Contributors

• @sfc-gh-rcheng made their first contribution in #345
• @sfc-gh-dwang made their first contribution in #352

Full Changelog: v1.1.0...v1.1.1

v1.1.0

This is the first non-beta release after v0.10.8 and it contains the support to do Snowpipe Streaming

What's Changed

• Rename file after we used gh actions to pass deploy script by @sfc-gh-japatel in #93
• new semgrep integration by @sfc-gh-spandey in #95
• PRODSEC-1059 integrate whitesource for github by @sfc-gh-jfan in #97
• Remove guava dependency by @sfc-gh-japatel in #101
• PRODSEC-1192 add jira flow by @sfc-gh-jfan in #103
• SNOW-500221 Client Api SDK changes by @sfc-gh-anwang in #102
• Add interface for insertFiles with clientInfo - Along with tests by @sfc-gh-japatel in #105
• SNOW-488437 Add connection timeout by @sfc-gh-anwang in #104
• SNOW-514117: Add illegal argument exception for negative client sequencer by @sfc-gh-anwang in #106
• Release a beta version of SDK for new Snowpipe APIs by @sfc-gh-japatel in #107
• Snowpipe Streaming V1 by @sfc-gh-tzhang in #109
• Add nullability to the field metadata by @sfc-gh-mnaides in #111
• SNOW-356153 Move streaming ingest to use profile.json by @sfc-gh-mnaides in #112
• Validate JSON for Object and String Date parsing by @sfc-gh-mnaides in #113
• SNOW-533760: include client name and key during upload by @sfc-gh-tzhang in #114
• SNOW-534044 validate max object and variant sizes by @sfc-gh-mnaides in #115
• NO-SNOW: small fixes on configuration file behaviors by @sfc-gh-tzhang in #116
• SNOW-535803 Support TIMESTAMP_TZ in streaming ingest by @sfc-gh-mnaides in #122
• NO-SNOW: update dependency by @sfc-gh-tzhang in #123
• NO-SNOW: simplify SnowflakeStreamingIngestClient interface by @sfc-gh-tzhang in #117
• NO_SNOW: SDK beta release 0.10.5-beta with streaming support by @sfc-gh-tzhang in #125
• SNOW-518618: remove BC-FIPS dependency by @sfc-gh-tzhang in #124
• NO-SNOW: Support running unit tests when profile.json is missing by @sfc-gh-tzhang in #127
• SNOW-540567: Disable test to unblock beta release by @sfc-gh-tzhang in #128
• SNOW-540821 Fetch signed url on every put call for gcs by @sfc-gh-mnaides in #129
• Allow client to configure parameters by @sfc-gh-mnaides in #126
• SNOW-534105: Enable encryption by default by @sfc-gh-tzhang in #130
• Pass in row index corresponding to input rows in response by @sfc-gh-japatel in #131
• Release version 1.0.0-beta by @sfc-gh-japatel in #132
• SNOW-350701 Refresh credentials on correct error by @sfc-gh-mnaides in #133
• SNOW-515458 Fix min/max for binary datatype by @sfc-gh-mnaides in #134
• SNOW-552292 fix max string ep info when the string is truncated by @sfc-gh-mnaides in #135
• Increase the default throttling threshold by @sfc-gh-mnaides in #138
• NO-SNOW: multiple small adjustments/fixes for Snowpipe Streaming by @sfc-gh-tzhang in #136
• NO-SNOW: adding more logging + fix IT tests by @sfc-gh-tzhang in #139
• Add row sequencer for debugging by @sfc-gh-japatel in #141
• SNOW-556588 Make timestamp test more robust by @sfc-gh-mnaides in #142
• Debug submit flush task only when isNeedFlush is true to avoid loggin… by @sfc-gh-japatel in #137
• Release version 1.0.1-beta by @sfc-gh-tzhang in #143
• @snow SNOW-554959 Align chunk encryption with block size and count for the whole interleaved file by @sfc-gh-azagrebin in #140
• SNOW-557106 enable jmx metrics for streaming by @sfc-gh-japatel in #144
• NO-SNOW: update code owner by @sfc-gh-tzhang in #147
• SNOW-560665: remove usage of instanceof for type checking by @sfc-gh-tzhang in #148
• SNOW-564031: Incorrect EP for NULL columns causing BDEC registration failure by @sfc-gh-tzhang in #149
• SNOW-470176 Update dependency com.fasterxml.jackson.core:jackson-databind to v2.13.2.1 by @mend-for-github-com in #151
• nosnow- support string date formats by @sfc-gh-mnaides in #153
• SNOW-566797: Support SDK with proxy server by @sfc-gh-tzhang in #154
• Release version 1.0.2-beta by @sfc-gh-tzhang in #150
• Enable ignored tests which run against sfctest0 prod1 account by @sfc-gh-japatel in #108
• no-snow fix error message printing by @sfc-gh-japatel in #157
• @snow SNOW-572229 Streaning Ingest: Manual padding of compressed chunk for encryption by @sfc-gh-azagrebin in #159
• NO-SNOW: add retry logic on 429 (TOO_MANY_REQUESTS) by @sfc-gh-tzhang in #160
• NO-SNOW: fix regression after the JMX metrics change by @sfc-gh-tzhang in #161
• NO-SNOW: Format code by @sfc-gh-kbregula in #155
• SNOW-556750 retry certain streaming ingest specific retries by @sfc-gh-mnaides in #158
• Use CloseableHttpClient and consume Entity on all APIs by @sfc-gh-japatel in #167
• NO_SNOW: fix bug in channel.close by @sfc-gh-tzhang in #163
• NO-SNOW: increase retry count due to server side Commit Service not making progress by @sfc-gh-tzhang in #172
• Fix logging project properties load exception by @sfc-gh-azagrebin in #171
• Release version 1.0.2-beta.1 by @sfc-gh-tzhang in #169
• NO-SNOW: accept account names with hyphen by @sfc-gh-tzhang in #174
• NO-SNOW: A few small fixes on Snowpipe Streaming PrPr by @sfc-gh-tzhang in #175
• no-snow Add readme and call offsetToken API in Example file by @sfc-gh-japatel in #177
• NO-SNOW: Update Arrow library to 7.0.0 to support VectorSchemaRootAppender by @sfc-gh-tzhang in #180
• SNOW-592855, SNOW-606515: fix channel.close time out issue due to high upload latency by @sfc-gh-tzhang in #182
• NO-SNOW: fix test failure due to time zone difference by @sfc-gh-tzhang in #183
• NO-SNOW: debug memory issue by @sfc-gh-tzhang in #185
• PRODSEC-1251 Whitesource Transition to Snyk by @sfc-gh-jfan in #170
• SNOW-610062: use "/" as name separator instead of using Paths.get by @sfc-gh-tzhang in #187
• SNOW-608483 fix column stats for collated strings by @sfc-gh-mnaides in #186
• NO-SNOW: improve README and Snowpipe Streaming Example by @sfc-gh-tzhang in #189
• V1.0.2-beta.2 release by @sfc-gh-tzhang in #190
• prodsec-1251 minor update on jira token by @sfc-gh-jfan in https://...

v1.0.3-beta.1

The 1.0.3-beta release has an issue with GCS on JDBC 3.13.18, so we revert #334 and then create a new release on top

What's Changed

• SNOW-663704 Send hex-encoded min/max for BINARY by @sfc-gh-lsembera in #329
• Remove JAXB by @sfc-gh-lsembera in #332
• NO-SNOW: revert https://github.com/snowflakedb/snowflake-kafka-connector/pull/546/files as it's causing failure on GCS by @sfc-gh-tzhang in #334
• V1.0.3-beta.1 release by @sfc-gh-tzhang in #335

Full Changelog: v1.0.3-beta...v1.0.3-beta.1

v1.0.3-beta

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] In order to increase timestamp parsing performance, the number of accepted timestamp formats has been limited. Please see the Javadoc for SnowflakeStreamingIngestChannel#insertRows for more details.
• [Improvement] Add column name to data validation error messages
• [Improvement] Convert empty variant strings into variant column to NULL
• [Improvement] Avoid datetime parsing wherever possible to improve performance
• [Improvement] Allow inf, -inf and nan strings into FLOAT columns
• [Improvement] Add telemetry to track the end2end latency
• [Fixes] Fix small parquet binary max length inconsistency with Arrow
• [Improvement] Add option to use parquet file writer per channel and then merge them into one per chunk
• [Improvement] Improve build latency for high throughput case with many channels
• [Fix] Fix various issues with unicode strings
• [Fix] Ingest SDK Does Not Honor http.nonProxyHosts JVM Argument
• [Improvement] Enable Parquet file by default and enforce single rowcount Parquet files

This release contains a few bug fixes and improvements for Snowpipe:

• [Fix] SimpleIngestIT runtime created database

What's Changed

• SNOW-710291 Allow inf, -inf and nan strings into FLOAT columns by @sfc-gh-lsembera in #306
• SNOW-684267 Add column name to data validation error messages by @sfc-gh-asen in #309
• SNOW-711338 Convert empty variant strings into Variant column to NULL by @sfc-gh-asen in #310
• SNOW-710401 Avoid datetime parsing wherever possible by @sfc-gh-lsembera in #311
• SNOW-701710: [Client side] Add telemetry to track the end2end latency for every BDEC by @sfc-gh-tzhang in #295
• SNOW-690793 use parquet file writer per channel and then merge them i… by @sfc-gh-gdoci in #299
• @no-snow fix small parquet binary max length inconsistency with Arrow by @sfc-gh-gdoci in #319
• SNOW-715618: Improve build latency for high throughput case with many channels by @sfc-gh-tzhang in #321
• SNOW-663621 Fix problems with unicode strings by @sfc-gh-lsembera in #300
• Enable Parquet data type tests by @sfc-gh-lsembera in #313
• [SNOW-726993] Ingest SDK Does Not Honor http.nonProxyHosts JVM Argument by @sfc-gh-wfateem in #325
• SNOW-731571 SimpleIngestIT runtime created database by @sfc-gh-lozhang in #327
• SNOW-620624 enforce single rowcount Parquet files by @sfc-gh-gdoci in #324
• SNOW-733699 make Parquet the default blob format version instead of Arrow by @sfc-gh-gdoci in #322
• SNOW-711299 New date/time processing by @sfc-gh-lsembera in #320
• NO-SNOW: various fixes/improvements in the SDK by @sfc-gh-tzhang in #330
• V1.0.3-beta release by @sfc-gh-tzhang in #326

New Contributors

• @sfc-gh-asen made their first contribution in #309
• @sfc-gh-wfateem made their first contribution in #325
• @sfc-gh-lozhang made their first contribution in #327

Full Changelog: v1.0.2-beta.7...v1.0.3-beta

v1.0.2-beta.7

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix incorrect behaviors for table with quoted columns
• [Bug Fix] Fix binary data type for Parquet
• [Bug Fix] Fix incorrect behaviors when sending telemetry data to Snowflake
• [Bug Fix] Fix incorrect behaviors for TIMESTAMP fractional part
• [Improvement] Remove circular dependency on channel object
• [Improvement] Add filename in Parquet MD to support

What's Changed

• SNOW-637237 Test opening several thousand channels in parallel by @sfc-gh-lsembera in #294
• @snow SNOW-657667 Streaming ingest: factor out channel from the buffer by @sfc-gh-azagrebin in #285
• SNOW-696155 fix binary values for Parquet by @sfc-gh-gdoci in #292
• SNOW-647377 Support quoted column names by @sfc-gh-lsembera in #293
• SNOW-703928 add two system properties for IT tests by @sfc-gh-gdoci in #296
• @snow SNOW-705508 Exclude slf4j-reload4j from Hadoop dep by @sfc-gh-azagrebin in #301
• NO-SNOW: fix client telemetry logic by @sfc-gh-tzhang in #297
• SNOW-706682 Log retry-related messages as INFO by @sfc-gh-lsembera in #302
• SNOW-705594: fix ConcurrentModificationException on the telemetry hashmap by @sfc-gh-tzhang in #303
• V1.0.2-beta.7 release by @sfc-gh-tzhang in #305
• SNOW-707520 Fix incorrect fractional part calculation of TIMESTAMP by @sfc-gh-lsembera in #304
• SNOW-704566 Add filename in Parquet extra MD by @sfc-gh-kkloudas in #298

Full Changelog: v1.0.2-beta.6...v1.0.2-beta.7

v1.0.2-beta.6

This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Improvement] Add parquet file support, this will be our default file format in the future
• [Improvement] Add support to specify a memory limit per client instead of using the system max memory
• [Improvement] Remove a bunch of unused code and setup code coverage

This release contains a few bug fixes and improvements for both Snowpipe and Snowpipe Streaming:

• [Improvement] Improve the HTTP retry logic

What's Changed

• SNOW-470176: [Snyk] Security upgrade com.fasterxml.jackson.core:jackson-databind from 2.13.2.1 to 2.13.4 by @sfc-gh-snyk-sca-sa in #237
• @snow SNOW-633432 Parquet file generator by @sfc-gh-azagrebin in #200
• SNOW-667402:Semgrep workflow update by @sfc-gh-hpathak in #234
• NO-SNOW: add support to reset a rowIndex in the error response whenever needed by @sfc-gh-tzhang in #241
• Revert "@snow SNOW-633432 Parquet file generator (#200)" by @sfc-gh-gdoci in #249
• Introduce parquet file generator by @sfc-gh-gdoci in #251
• Revert "Introduce parquet file generator (#251)" by @sfc-gh-tzhang in #269
• [Snyk] Security upgrade org.apache.arrow:arrow-compression from 9.0.0 to 10.0.0 by @snyk-bot in #271
• SNOW-637237 Thread-safe HTTP client initialisation by @sfc-gh-lsembera in #277
• SNOW-684474: Support adding a memory limit through the config file by @sfc-gh-tzhang in #279
• @snow SNOW-633432 Parquet file generator (fixed dep) by @sfc-gh-azagrebin in #280
• SNOW-637237 Retry for 2 minutes since the first failure by @sfc-gh-lsembera in #281
• Revert "@snow SNOW-612516 Streaming ingest: add random file reader to scan single columns, format version 2 by @sfc-gh-azagrebin in #282
• NO-SNOW: enable code coverage check in the SDK by @sfc-gh-tzhang in #283
• @snow no-snow Streaming ingest: always send BDEC version as server side supports this by @sfc-gh-azagrebin in #284
• @snow no-snow Fix more parquet dependencies causing vulnerabilities by @sfc-gh-azagrebin in #287
• V1.0.2-beta.6 release by @sfc-gh-tzhang in #286

New Contributors

• @sfc-gh-hpathak made their first contribution in #234
• @snyk-bot made their first contribution in #271

Full Changelog: v1.0.2-beta.5...v1.0.2-beta.6

v1.0.2-beta.5

Please Note: This is a release on top of v1.0.2-beta.4
This release contains a few bug fixes and improvements for Snowpipe Streaming

[Improvement] Improve and fix a few data type validation logic
[Improvement] Add throttling logic on direct memory based on uploading queue size
[Improvement] Add basic logic to support Parquet file format, this work is still in progress and we're still using Arrow
[Improvement] Improve exception handling logic

What's Changed

• PRODSEC-1924 rm whitesource legacy files by @sfc-gh-jfan in #217
• @snow SNOW-657206 SteamingIngestIT: add random suffix to DB name to fix concurrent CI runs by @sfc-gh-azagrebin in #222
• @snow SNOW-633432 Refactor RowBuffer and Flusher API by @sfc-gh-azagrebin in #202
• @snow SNOW-633432 Refactor tests to support various BDEC format versions by @sfc-gh-azagrebin in #209
• SNOW-470176: [Snyk] Fix for 2 vulnerabilities by @sfc-gh-snyk-sca-sa in #225
• SNOW-655614 Data type validation: Boolean by @sfc-gh-lsembera in #228
• SNOW-655614 Data type validation: Date and Time by @sfc-gh-lsembera in #229
• SNOW-655614 Data type validation: NUMBER and REAL by @sfc-gh-lsembera in #230
• SNOW-655614 Data type validation: String and Binary by @sfc-gh-lsembera in #231
• SNOW-616916: throttle the insertRows API when the flush task queue size reaches a certain threshold by @sfc-gh-tzhang in #215
• NO-SNOW: add stacktrace during blob building failure by @sfc-gh-tzhang in #232
• SNOW-655614 Data type validation: Semi-structured types by @sfc-gh-lsembera in #233
• V1.0.2-beta.5 release by @sfc-gh-tzhang in #235

New Contributors

• @sfc-gh-snyk-sca-sa made their first contribution in #225
• @sfc-gh-lsembera made their first contribution in #228

Full Changelog: v1.0.2-beta.4...v1.0.2-beta.5

v1.0.2-beta.4

Please Note: This is a release on top of v1.0.2-beta.3
This release contains a few bug fixes and improvements for Snowpipe Streaming

• [Bug Fix] Check max column length for Binary data type
• [Bug Fix] Fix ARRAY data type with String elements
• [Improvement] Support running IT tests as part of merge gate
• [Improvement] Update the InsertError response to include extra or missing column information

What's Changed

• Run streaming ingest integration tests on build by @sfc-gh-mnaides in #207
• SNOW-644959: Return all the extra columns in the insertRow response by @sfc-gh-tzhang in #210
• SNOW-637927: Check max column length for Binary data type by @sfc-gh-tzhang in #206
• SNOW-649753: Update InsertError to include extra/missing column info for KC schema evolution by @sfc-gh-tzhang in #212
• SNOW-644956: fix string data type in ARRAY by @sfc-gh-tzhang in #211
• V1.0.2-beta.4 release by @sfc-gh-tzhang in #213

Full Changelog: v1.0.2-beta.3...v1.0.2-beta.4

v1.0.2-beta.3

Please Note: This is a release on top of v1.0.2-beta.2
This release contains a few bug fixes and improvements for Snowpipe Streaming:

• [Bug Fix] Fix default min/max value in EP for REAL column
• [Bug Fix] Fix register retry logic
• [Bug Fix] Support ARRAY data type
• [Bug Fix] Fix timestamp used when generating the blob
• [Improvement] Support column level compression of the blob (not enabled yet)
• [Improvement] Support client telemetry

This release contains a few bug fixes and improvements for both Snowpipe and Snowpipe Streaming:

• [Improvement] Add HTTP connection pooling manager for better HTTP connection management

What's Changed

• V1.0.2-beta.2 release by @sfc-gh-tzhang in #190
• prodsec-1251 minor update on jira token by @sfc-gh-jfan in #192
• SNOW-619777, SNOW-606454: fix default min/max value in EP for real column by @sfc-gh-tzhang in #194
• NO-SNOW http connection pooling manager by @sfc-gh-japatel in #193
• NO-SNOW check the retry logic with millis instead of ms and seconds by @sfc-gh-japatel in #197
• SNOW-626104: Support ARRAY data type in the SDK by @sfc-gh-tzhang in #196
• NO-SNOW: fix logging format to NOT add a new line by @sfc-gh-tzhang in #198
• @snow SNOW-612516 Streaming ingest: add random file reader to scan single columns, format version 2 by @sfc-gh-azagrebin in #191
• SNOW-356156: support client telemetry through client_telemetry_v by @sfc-gh-tzhang in #195
• @snow SNOW-634840 Disable sending blob_version to server side until supported in prod by @sfc-gh-azagrebin in #201
• SNOW-593873 SNOW-636695 Round timestamp to seconds when creating blob name by @sfc-gh-kkloudas in #203
• V1.0.2-beta.3 release by @sfc-gh-tzhang in #204

New Contributors

• @sfc-gh-kkloudas made their first contribution in #203

Full Changelog: v1.0.2-beta.2...v1.0.2-beta.3

v0.10.8

Fix on top of v0.10.7

• Use this version if you are considering using 0.10.7
• All fixes from 0.10.7 plus a fixes a bug in it
  • Fix IllegalStateException found while testing