Skip to content

Latest commit

 

History

History
115 lines (87 loc) · 7.75 KB

README.md

File metadata and controls

115 lines (87 loc) · 7.75 KB
Raw

Native SDK for Connectors

Data is fuel for data clouds. The journey of data begins with transferring it into Snowflake, because of that this is essential step for customers.

There are many various ways of importing the data to the Snowflake environment. One of them is by using connectors. A connector is an application that allows data flow from an external source system into Snowflake. A Native Connector is a connector built on the foundations of the Snowflake Native App Framework.

In order to accelerate and standardize the process of developing new Native Connectors, Snowflake would like to introduce the Native SDK for Connectors which is a set of libraries, developing tools, example connectors and connector template projects.

Key components

In the diagram below, you can find key components that are listed with a brief description below the diagram.

Repository Components Diagram

Native SDK for Connectors Java

Native SDK for Connectors Java is a library that is distributed through the Maven Central Repository. The library significantly helps developers in building Snowflake Native Applications of connectors' type. The SDK consists of both Java and sql components. Sql components can be found in the .sql files placed in the resources directory. They provide the definitions of the database objects, that are created during the installation of the Connector. Java components mainly serve as procedure handlers. However, among these Java classes, there also are some helper/util classes that provide useful tools to tackle most common use cases. The database objects and Java classes create a coherent whole that makes managing the state of the Connector much easier and lets the developers focus on the implementation of the specific external source ingestion logic which also is easier to do with the SDK.

Reach the official documentation and java docs to learn more.

Task Reactor

This major component is built into Native SDK for Connectors Java library. It consists of .sql files and Java classes embedded into the library that helps in managing and executing work asynchronously, mostly the work related to data ingestion. This component offers stability and scalability in work executing. Although this component is dedicated to orchestrating the ingestion tasks, it also can be used for executing non-ingestion tasks that require an asynchronous way of the execution.

Learn more from the official documentation

Native SDK for Connectors Java Test

A separate Java library distributed through the Maven Central Repository which consists of Java utils helping in testing the developed Connectors. These utils include custom assertions and in-memory objects (mocks) for Java classes used in the Native SDK for Connectors Java library.

Connectors Native SDK Template

The template Gradle Java project with inbuilt Native Java SDK that allows the developer to deploy, install, and run the sample, mocked source connector right after downloading the template. The template is filled with some code already which shows how to use the Native SDK for Connectors Java according to the connector flow defined by the Native SDK for Connectors. Reach the official tutorial that will guide through the whole developer flow starting from cloning the template project, through the implementation process of key functionalities, ending with the deployed and running connector in the Snowflake environment!

Example connectors

These are example projects of connectors that provide the general information on how the connectors, as Native Applications, should be created and deployed to the Snowflake environment. Among these connectors there is the one that is built on the top of the Connectors Native SDK Template project, to show in practice how the template should be used to develop a new connector project. There are also some basic examples of connectors that show the general concept of the native connectors:

  • push-based connectors
  • basic pull-based connectors written in Python
  • basic pull-based connectors written in Java

Keep in mind that these basic connectors do not use the Native SDK for Connectors Java library.

Configuring Snowflake CLI for templates and examples

In order to use the templates and examples, it is required to configure the Snowflake CLI. The configuration file can be found in root directory of the repository. The file is named Snowflake CLI configuration file path ./deployment/snowflake.toml The file should be fulfilled with the required data depending on users Snowflake credentials. More about configuring that file can be found in the [Snowflake CLI documentation] Snowflake CLI docs.

Structure of the repository

Contributing

Please refer to CONTRIBUTING.md.

License

Please refer to LICENSE