Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1798694: Improve SDK local setup documentation to enable logging #934

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

SNOW-1798694: Improve SDK local setup documentation to enable logging #934

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a53180c
update readme
sfc-gh-tzhang Jan 24, 2025
1480525
format
sfc-gh-tzhang Jan 24, 2025
4d5ea58
remove
sfc-gh-tzhang Jan 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 47 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@ Snowflake Ingest Service Java SDK
The Snowflake Ingest Service SDK allows users to ingest files into their
Snowflake data warehouse in a programmatic fashion via key-pair
authentication. Currently, we support ingestion through the following APIs:

1. [Snowpipe](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-gs.html#client-requirement-java-or-python-sdk)
2. [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview)

# Dependencies

The Snowflake Ingest Service SDK depends on the following libraries:

* snowflake-jdbc (3.16.1+)
* snowflake-jdbc (3.16.1+)
* slf4j-api
* com.github.luben:zstd-jni (1.5.0-1)

Expand Down Expand Up @@ -63,13 +64,17 @@ dependencies {

## Jar Versions

The Snowflake Ingest SDK provides shaded and unshaded versions of its jar. The shaded version bundles the dependencies into its own jar,
whereas the unshaded version declares its dependencies in `pom.xml`, which are fetched as standard transitive dependencies by the build system like Maven or Gradle.
The shaded JAR can help avoid potential dependency conflicts, but the unshaded version provides finer graned control over transitive dependencies.
The Snowflake Ingest SDK provides shaded and unshaded versions of its jar. The shaded version bundles the dependencies
into its own jar,
whereas the unshaded version declares its dependencies in `pom.xml`, which are fetched as standard transitive
Comment on lines +68 to +69
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, I think these lines should be combined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they're combined, you can do "View File" and it will show you a better version

dependencies by the build system like Maven or Gradle.
The shaded JAR can help avoid potential dependency conflicts, but the unshaded version provides finer graned control
over transitive dependencies.

## Using with snowflake-jdbc-fips

For use cases, which need to use `snowflake-jdbc-fips` instead of the default `snowflake-jdbc`, we recommend to take the following steps:
For use cases, which need to use `snowflake-jdbc-fips` instead of the default `snowflake-jdbc`, we recommend to take the
following steps:

- Use the unshaded version of the Ingest SDK.
- Exclude these transitive dependencies:
Expand All @@ -78,7 +83,8 @@ For use cases, which need to use `snowflake-jdbc-fips` instead of the default `s
- `org.bouncycastle:bcprov-jdk18on`
- Add a dependency on `snowflake-jdbc-fips`.

See [this test](https://github.com/snowflakedb/snowflake-ingest-java/tree/master/e2e-jar-test/fips) for an example how to use Snowflake Ingest SDK together with Snowflake FIPS JDBC Driver.
See [this test](https://github.com/snowflakedb/snowflake-ingest-java/tree/master/e2e-jar-test/fips) for an example how
to use Snowflake Ingest SDK together with Snowflake FIPS JDBC Driver.

# Example

Expand All @@ -89,16 +95,26 @@ Check out `SnowflakeIngestBasicExample.java`
## Snowpipe Streaming

Check out `SnowflakeStreamingIngestExample.java`, which performs following operations:
1. Reads a JSON file which contains details regarding Snowflake Account, User, Role and Private Key. Take a look at `profile_streaming.json.example` for more details.
1. [Here](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) are the steps required to generate a private key.
2. Creates a `SnowflakeStreamingIngestClient` which can be used to open one or more Streaming Channels pointing to the same or different tables.

1. Reads a JSON file which contains details regarding Snowflake Account, User, Role and Private Key. Take a look at
`profile_streaming.json.example` for more details.
1. [Here](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) are the
steps required to generate a private key.
2. Creates a `SnowflakeStreamingIngestClient` which can be used to open one or more Streaming Channels pointing to the
same or different tables.
3. Creates a `SnowflakeStreamingIngestChannel` against a Database, Schema and Table.
1. Please note: The database, schema and table is expected to be present before opening the Channel. Example SQL queries to create them:
1. Please note: The database, schema and table is expected to be present before opening the Channel. Example SQL
queries to create them:

```sql
create or replace database MY_DATABASE;
create or replace schema MY_SCHEMA;
create or replace table MY_TABLE(c1 number);
create
or replace database MY_DATABASE;
create
or replace schema MY_SCHEMA;
create
or replace table MY_TABLE(c1 number);
Comment on lines +110 to +115
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The formatting here seems wrong

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is weird, let me see why auto formatter decided to do this

```

4. Inserts 1000 rows into the channel created in 3rd step using the `insertRows` API on the Channel object
1. `insertRows` API also takes in an optional `offsetToken` String which can be associated to this batch of rows.
5. Calls `getLatestCommittedOffsetToken` on the channel until the appropriate offset is found in Snowflake.
Expand All @@ -123,28 +139,34 @@ mvn package
However, for general usage, pulling a pre-built jar from maven is
recommended.

If you would like to run SnowflakeIngestBasicExample.java or SnowflakeStreamingIngestExample.java in the example folder,
If you would like to run SnowflakeIngestBasicExample.java or SnowflakeStreamingIngestExample.java in the example folder,
please edit `pom.xml` and change the scope of the dependency `slf4j-simple` from `test` to `runtime` in order to enable
console log output.

console log output. For changing the logging level or message format, updating the system property or a
simplelogger.properties file on the classpath is required, please
see https://www.slf4j.org/api/org/slf4j/simple/SimpleLogger.html for more details. If you run into formatting errors,
please run format.sh to format the code.
Comment on lines +144 to +147
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the major changes, all other changes in this file are done by the auto formatter


# Testing (SimpleIngestIT Test)

- Modify `TestUtils.java` file and replace *PROFILE_PATH* with `profile.json.example` for testing.
- Modify `TestUtils.java` file and replace *PROFILE_PATH* with `profile.json.example` for testing.

- `profile.json` is used because an encrypted file will be
decrypted for Github Actions testing. Check `End2EndTest.yml`
- `profile.json` is used because an encrypted file will be
decrypted for Github Actions testing. Check `End2EndTest.yml`

- Use an unencrypted version(Only for testing) of private key while generating keys(private and public pair) using OpenSSL.
- Use an unencrypted version(Only for testing) of private key while generating keys(private and public pair) using
OpenSSL.

- Here is the link for documentation [Key Pair
Generator](https://docs.snowflake.com/en/user-guide/key-pair-auth.html)
- Here is the link for documentation [Key Pair
Generator](https://docs.snowflake.com/en/user-guide/key-pair-auth.html)

# Contributing to this repo

Each PR must pass all required github action merge gates before approval and merge. In addition to those tests, you will need:
Each PR must pass all required github action merge gates before approval and merge. In addition to those tests, you will
need:

- Formatter: run this script [`./format.sh`](https://github.com/snowflakedb/snowflake-ingest-java/blob/master/format.sh) from root
- CLA: all contributers must sign the Snowflake CLA. This is a one time signature, please provide your email so we can work with you to get this signed after you open a PR.
- Formatter: run this script [`./format.sh`](https://github.com/snowflakedb/snowflake-ingest-java/blob/master/format.sh)
from root
- CLA: all contributers must sign the Snowflake CLA. This is a one time signature, please provide your email so we can
work with you to get this signed after you open a PR.

Thank you for contributing! We will review and approve PRs as soon as we can.
13 changes: 6 additions & 7 deletions .../java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,14 +241,13 @@ public SnowflakeStreamingIngestClientInternal(

this.snowflakeServiceClient = new SnowflakeServiceClient(this.httpClient, this.requestBuilder);

this.storageManager =
parameterProvider.isEnableIcebergStreaming()
? new SubscopedTokenExternalVolumeManager(
this.role, this.name, this.snowflakeServiceClient)
: new InternalStageManager(
isTestMode, this.role, this.name, this.snowflakeServiceClient);

try {
this.storageManager =
parameterProvider.isEnableIcebergStreaming()
? new SubscopedTokenExternalVolumeManager(
this.role, this.name, this.snowflakeServiceClient)
: new InternalStageManager(
isTestMode, this.role, this.name, this.snowflakeServiceClient);
this.flushService =
new FlushService<>(this, this.channelCache, this.storageManager, this.isTestMode);
} catch (Exception e) {
Expand Down
Loading