diff --git a/README.md b/README.md new file mode 100644 index 0000000..a413373 --- /dev/null +++ b/README.md @@ -0,0 +1,97 @@ +# SingleStore Fivetran Connector + +## Pre-requisites for development + +- JDK v17 +- Gradle 8 ([here](https://gradle.org/install/#manually) is an installation instruction) + +## Steps for starting server + +1. Build the Jar + +``` +gradle jar +``` + +2. Run the Jar + +``` +java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar +``` + +## Steps for running Java tests + +1. Start SingleStore cluster + +``` +docker run \ + -d --name singlestoredb-dev \ + -e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \ + -e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \ + -e SINGLESTORE_VERSION="8.7.16" \ + -p 3306:3306 -p 8080:8080 -p 9000:9000 \ + ghcr.io/singlestore-labs/singlestoredb-dev:latest +``` + +2. Create `ROOT_PASSWORD` environment variable + +``` +export ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" +``` + +3. Run tests + +``` +gradle build +``` + +## Steps for using Connector tester + +1. Start SingleStore cluster + +``` +docker run \ + -d --name singlestoredb-dev \ + -e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \ + -e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \ + -e SINGLESTORE_VERSION="8.7.16" \ + -p 3306:3306 -p 8080:8080 -p 9000:9000 \ + ghcr.io/singlestore-labs/singlestoredb-dev:latest +``` + +2. Create database and table + +``` +DROP DATABASE IF EXISTS tester; +CREATE DATABASE tester; +USE tester; +CREATE TABLE t(a INT PRIMARY KEY, b INT); +``` + +3. Start Connector server + +``` +gradle jar +java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar +``` + +4. Run the tester by following instructions + from [here](https://github.com/fivetran/fivetran_sdk/blob/main/tools/destination-tester/README.md). + As a command use you can use + +``` +docker run --mount type=bind,source=./tester,target=/data -a STDIN -a STDOUT -a STDERR -it -e GRPC_HOSTNAME=localhost --network=host fivetrandocker/fivetran-sdk-tester:0.24.0729.001 --tester-type source --port 55051 +``` + +5. Update table + +``` +INSERT INTO t VALUES(1, 2); +INSERT INTO t VALUES(2, 2); +DELETE FROM t WHERE a = 1; +UPDATE t SET b = 3 WHERE a = 2; +``` + +6. Check the content of `./tester/warehouse.db` file + using [DuckDB](https://duckdb.org/docs/api/cli/overview.html) CLI + or [DBeaver](https://duckdb.org/docs/guides/sql_editors/dbeaver) \ No newline at end of file diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 0000000..4d4cef1 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,110 @@ +--- +name: SingleStore +title: SingleStore connector by Fivetran | Fivetran documentation +Description: Connect your SingleStore data to your destination using Fivetran. +hidden: false +--- + +# SingleStore {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%} + +[SingleStore](https://www.singlestore.com/) is a distributed, cloud-native database that can handle +transactional and analytical workloads with a unified engine. It provides real-time analytics, +transactions, and streaming capabilities, enabling users to handle diverse workloads on a single +platform. + +------------------ + +## Features + +{% featureTable connector="singlestore" %} +Capture Deletes: +Custom Data: +{% /featureTable %} + +------------------ + +## Setup guide + +Follow our [step-by-step SingleStore setup guide](/docs/{path}/setup-guide) to connect +SingleStore with your destination using Fivetran connectors. + +------------------ + +## Sync overview + +Once Fivetran is connected to your SingleStore deployment, the connector fetches an initial +consistent snapshot of all data from your SingleStore table. Once the initial sync is complete, the +connector stream UPDATE/DELETE/INSERT events made to your SingleStore table. + +SingleStore connector +uses [OBSERVE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/observe/) +to capture change events. + +Connector does not support handling schema changes. You cannot run `ALTER` and `DROP` queries +while the `OBSERVE` query is running. + +------------------ + +## Schema information + +Fivetran replicates a single table of the SingleStore database. Selected SingleStore database is +mapped to Fivetran schema. + +### Fivetran-generated columns + +Fivetran adds the following columns to table in your destination: + +- `_fivetran_deleted` (BOOLEAN) marks deleted rows in the source database. +- `_fivetran_synced` (UTC TIMESTAMP) indicates when Fivetran last successfully synced the row. +- `_fivetran_index` (INTEGER) shows the order of updates for tables that do not have a primary key. +- `_fivetran_id` (STRING) is the hash of the non-Fivetran values of each row. It's a unique ID that + Fivetran uses to avoid duplicate rows in tables that do not have a primary key. + +### Type transformations and mapping + +As we extract your data, we match SingleStore data types in your SingleStore database to types that +Fivetran supports. If we don't support a specific data type, we automatically change that type to +the closest supported type. + +The following table illustrates how we transform your SingleStore data types into Fivetran supported +types: + +| SingleStore Data Type | Fivetran Data Type | Notes | +|-----------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------| +| BOOL | SHORT | +| BIT | BINARY | +| TINYINT | SHORT | +| SMALLINT | SHORT | +| MEDIUMINT | INT | +| INT | INT | +| BIGINT | LONG | +| FLOAT | FLOAT | +| DOUBLE | DOUBLE | +| DECIMAL | DECIMAL | +| DATE | NAIVE_DATE | +| TIME | NAIVE_DATETIME | +| TIME(6) | NAIVE_DATETIME | +| DATETIME | NAIVE_DATETIME | +| DATETIME(6) | NAIVE_DATETIME | +| TIMESTAMP | NAIVE_DATETIME | +| TIMESTAMP(6) | NAIVE_DATETIME | +| YEAR | NAIVE_DATE | +| CHAR | STRING | +| VARCHAR | STRING | +| TINYTEXT | STRING | +| TEXT | STRING | +| MEDIUMTEXT | STRING | +| LONGTEXT | STRING | +| BINARY | BINARY | +| VARBINARY | BINARY | +| TINYBLOB | BINARY | +| BLOB | BINARY | +| MEDIUMBLOB | BINARY | +| LONGBLOB | BINARY | +| JSON | JSON | +| BSON | BINARY | +| GEOGRAPHY | STRING | +| GEOGRAPHYPOINT | STRING | +| ENUM | STRING | +| SET | STRING | +| VECTOR | BINARY/JSON | If the `vector_type_project_format` variable is set to `BINARY` - `VECTOR` is mapped to `BINARY`, otherwise it is mapped to `JSON` | diff --git a/docs/setup-guide.md b/docs/setup-guide.md new file mode 100644 index 0000000..876d9a6 --- /dev/null +++ b/docs/setup-guide.md @@ -0,0 +1,101 @@ +--- +name: SingleStore +title: SingleStore source connector for Fivetran +description: Read step-by-step instructions on how to connect SingleStore with your destination using Fivetran connectors. +hidden: false +--- + +# SingleStore Setup Guide {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%} + +Follow our setup guide to connect SingleStore to Fivetran. + +> NOTE: This connector is [partner-built](/docs/partner-built-program). For any questions related +> to SingleStore connector and its documentation, contact SingleStore by raising an issue in the +> [SingleStore Fivetran Connector](https://github.com/singlestore-labs/singlestore-fivetran-connector) +> GitHub repository. + +----- + +## Prerequisites + +To connect your SingleStore database to Fivetran, you need: + +- A SingleStore instance of version 8.7.16 or higher. Refer + to [Creating and Using Workspaces](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/creating-and-using-workspaces/) + for instructions on creating a SingleStore workspace in + the [Cloud Portal](https://portal.singlestore.com/). + To deploy a Self-Managed cluster instead, refer + to [Deploy](https://docs.singlestore.com/db/latest/deploy/). Once the SingleStore + workspace/cluster is Active, you'll need the following to connect to Fivetran: + - `Host` + - `Port` + - `Username` + - `Password` + - `Database` + - `Table` +- A Fivetran account with + the [Connector Creator](/docs/using-fivetran/fivetran-dashboard/account-management/role-based-access-control#rbacpermissions) + permissions. + +--- + +## Setup instructions + +To authorize Fivetran to connect to your SinlgeStore database, follow these instructions: + +### Configure SingleStore + +1. Configure your firewall and/or other access control systems to allow incoming connections to your + SingleStore instance from [Fivetran's IPs](https://fivetran.com/docs/using-fivetran/ips) for your + region. +2. Ensure that the SingleStore database user has the `SELECT` permission. +3. Enable support of OBSERVE queries + +``` +SET enable_observe_queries=1 +``` + +### Finish Fivetran configuration + +1. Log in to your Fivetran account. +2. Go to the [**Connectors** page](https://fivetran.com/dashboard/connectors), and then click * + *+ Add connector**. +3. Select **SingleStore** as the connector type. +4. Enter the following connection configurations for you SingleStore workspace/cluster: + * **Host** + * **Port** + * **Database** + * **Table** + * **Username** + * **Password** +5. (Optional) Enable SSL and specify related configurations. +6. (Optional) Specify additional **Driver Parameters**. Refer + to [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#connection-string-parameters) + documentation for a list of supported parameters. +7. Click **Save & Test**. + +### Setup tests + +Fivetran performs the following SingleStore connection tests: + +- The Connection test checks if Fivetran can connect to your SingleStore cluster using credentials + provided in the setup form +- The Table test checks if specified table exists + +--- + +## Related articles + +[ Connector Overview](/docs/connectors/databases/singlestore) + + + +[ Schema Information](/docs/connectors/databases/singlestore#schemainformation) + + + +[ API Connector Configuration](/docs/rest-api/connectors/config#sinlgestore) + + + +[ Documentation Home](/docs/getting-started) \ No newline at end of file diff --git a/src/main/java/com/singlestore/fivetran/connector/SingleStoreConnectorServiceImpl.java b/src/main/java/com/singlestore/fivetran/connector/SingleStoreConnectorServiceImpl.java index c94d974..4a42128 100644 --- a/src/main/java/com/singlestore/fivetran/connector/SingleStoreConnectorServiceImpl.java +++ b/src/main/java/com/singlestore/fivetran/connector/SingleStoreConnectorServiceImpl.java @@ -145,7 +145,7 @@ public void update(UpdateRequest request, StreamObserver try { State state; - if (request.hasStateJson()) { + if (request.hasStateJson() && !request.getStateJson().equals("{}")) { state = State.fromJson(request.getStateJson()); } else { state = new State(conn.getNumPartitions()); diff --git a/tester/config.json b/tester/config.json new file mode 100644 index 0000000..7a73a41 --- /dev/null +++ b/tester/config.json @@ -0,0 +1,2 @@ +{ +} \ No newline at end of file