-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Adalbert Makarovych
committed
Oct 21, 2024
1 parent
2690fed
commit d74a7a5
Showing
5 changed files
with
311 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
# SingleStore Fivetran Connector | ||
|
||
## Pre-requisites for development | ||
|
||
- JDK v17 | ||
- Gradle 8 ([here](https://gradle.org/install/#manually) is an installation instruction) | ||
|
||
## Steps for starting server | ||
|
||
1. Build the Jar | ||
|
||
``` | ||
gradle jar | ||
``` | ||
|
||
2. Run the Jar | ||
|
||
``` | ||
java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar | ||
``` | ||
|
||
## Steps for running Java tests | ||
|
||
1. Start SingleStore cluster | ||
|
||
``` | ||
docker run \ | ||
-d --name singlestoredb-dev \ | ||
-e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \ | ||
-e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \ | ||
-e SINGLESTORE_VERSION="8.7.16" \ | ||
-p 3306:3306 -p 8080:8080 -p 9000:9000 \ | ||
ghcr.io/singlestore-labs/singlestoredb-dev:latest | ||
``` | ||
|
||
2. Create `ROOT_PASSWORD` environment variable | ||
|
||
``` | ||
export ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" | ||
``` | ||
|
||
3. Run tests | ||
|
||
``` | ||
gradle build | ||
``` | ||
|
||
## Steps for using Connector tester | ||
|
||
1. Start SingleStore cluster | ||
|
||
``` | ||
docker run \ | ||
-d --name singlestoredb-dev \ | ||
-e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \ | ||
-e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \ | ||
-e SINGLESTORE_VERSION="8.7.16" \ | ||
-p 3306:3306 -p 8080:8080 -p 9000:9000 \ | ||
ghcr.io/singlestore-labs/singlestoredb-dev:latest | ||
``` | ||
|
||
2. Create database and table | ||
|
||
``` | ||
DROP DATABASE IF EXISTS tester; | ||
CREATE DATABASE tester; | ||
USE tester; | ||
CREATE TABLE t(a INT PRIMARY KEY, b INT); | ||
``` | ||
|
||
3. Start Connector server | ||
|
||
``` | ||
gradle jar | ||
java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar | ||
``` | ||
|
||
4. Run the tester by following instructions | ||
from [here](https://github.com/fivetran/fivetran_sdk/blob/main/tools/destination-tester/README.md). | ||
As a command use you can use | ||
|
||
``` | ||
docker run --mount type=bind,source=./tester,target=/data -a STDIN -a STDOUT -a STDERR -it -e GRPC_HOSTNAME=localhost --network=host fivetrandocker/fivetran-sdk-tester:0.24.0729.001 --tester-type source --port 55051 | ||
``` | ||
|
||
5. Update table | ||
|
||
``` | ||
INSERT INTO t VALUES(1, 2); | ||
INSERT INTO t VALUES(2, 2); | ||
DELETE FROM t WHERE a = 1; | ||
UPDATE t SET b = 3 WHERE a = 2; | ||
``` | ||
|
||
6. Check the content of `./tester/warehouse.db` file | ||
using [DuckDB](https://duckdb.org/docs/api/cli/overview.html) CLI | ||
or [DBeaver](https://duckdb.org/docs/guides/sql_editors/dbeaver) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
--- | ||
name: SingleStore | ||
title: SingleStore connector by Fivetran | Fivetran documentation | ||
Description: Connect your SingleStore data to your destination using Fivetran. | ||
hidden: false | ||
--- | ||
|
||
# SingleStore {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%} | ||
|
||
[SingleStore](https://www.singlestore.com/) is a distributed, cloud-native database that can handle | ||
transactional and analytical workloads with a unified engine. It provides real-time analytics, | ||
transactions, and streaming capabilities, enabling users to handle diverse workloads on a single | ||
platform. | ||
|
||
------------------ | ||
|
||
## Features | ||
|
||
{% featureTable connector="singlestore" %} | ||
Capture Deletes: | ||
Custom Data: | ||
{% /featureTable %} | ||
|
||
------------------ | ||
|
||
## Setup guide | ||
|
||
Follow our [step-by-step SingleStore setup guide](/docs/{path}/setup-guide) to connect | ||
SingleStore with your destination using Fivetran connectors. | ||
|
||
------------------ | ||
|
||
## Sync overview | ||
|
||
Once Fivetran is connected to your SingleStore deployment, the connector fetches an initial | ||
consistent snapshot of all data from your SingleStore table. Once the initial sync is complete, the | ||
connector stream UPDATE/DELETE/INSERT events made to your SingleStore table. | ||
|
||
SingleStore connector | ||
uses [OBSERVE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/observe/) | ||
to capture change events. | ||
|
||
Connector does not support handling schema changes. You cannot run `ALTER` and `DROP` queries | ||
while the `OBSERVE` query is running. | ||
|
||
------------------ | ||
|
||
## Schema information | ||
|
||
Fivetran replicates a single table of the SingleStore database. Selected SingleStore database is | ||
mapped to Fivetran schema. | ||
|
||
### Fivetran-generated columns | ||
|
||
Fivetran adds the following columns to table in your destination: | ||
|
||
- `_fivetran_deleted` (BOOLEAN) marks deleted rows in the source database. | ||
- `_fivetran_synced` (UTC TIMESTAMP) indicates when Fivetran last successfully synced the row. | ||
- `_fivetran_index` (INTEGER) shows the order of updates for tables that do not have a primary key. | ||
- `_fivetran_id` (STRING) is the hash of the non-Fivetran values of each row. It's a unique ID that | ||
Fivetran uses to avoid duplicate rows in tables that do not have a primary key. | ||
|
||
### Type transformations and mapping | ||
|
||
As we extract your data, we match SingleStore data types in your SingleStore database to types that | ||
Fivetran supports. If we don't support a specific data type, we automatically change that type to | ||
the closest supported type. | ||
|
||
The following table illustrates how we transform your SingleStore data types into Fivetran supported | ||
types: | ||
|
||
| SingleStore Data Type | Fivetran Data Type | Notes | | ||
|-----------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------| | ||
| BOOL | SHORT | | ||
| BIT | BINARY | | ||
| TINYINT | SHORT | | ||
| SMALLINT | SHORT | | ||
| MEDIUMINT | INT | | ||
| INT | INT | | ||
| BIGINT | LONG | | ||
| FLOAT | FLOAT | | ||
| DOUBLE | DOUBLE | | ||
| DECIMAL | DECIMAL | | ||
| DATE | NAIVE_DATE | | ||
| TIME | NAIVE_DATETIME | | ||
| TIME(6) | NAIVE_DATETIME | | ||
| DATETIME | NAIVE_DATETIME | | ||
| DATETIME(6) | NAIVE_DATETIME | | ||
| TIMESTAMP | NAIVE_DATETIME | | ||
| TIMESTAMP(6) | NAIVE_DATETIME | | ||
| YEAR | NAIVE_DATE | | ||
| CHAR | STRING | | ||
| VARCHAR | STRING | | ||
| TINYTEXT | STRING | | ||
| TEXT | STRING | | ||
| MEDIUMTEXT | STRING | | ||
| LONGTEXT | STRING | | ||
| BINARY | BINARY | | ||
| VARBINARY | BINARY | | ||
| TINYBLOB | BINARY | | ||
| BLOB | BINARY | | ||
| MEDIUMBLOB | BINARY | | ||
| LONGBLOB | BINARY | | ||
| JSON | JSON | | ||
| BSON | BINARY | | ||
| GEOGRAPHY | STRING | | ||
| GEOGRAPHYPOINT | STRING | | ||
| ENUM | STRING | | ||
| SET | STRING | | ||
| VECTOR | BINARY/JSON | If the `vector_type_project_format` variable is set to `BINARY` - `VECTOR` is mapped to `BINARY`, otherwise it is mapped to `JSON` | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
--- | ||
name: SingleStore | ||
title: SingleStore source connector for Fivetran | ||
description: Read step-by-step instructions on how to connect SingleStore with your destination using Fivetran connectors. | ||
hidden: false | ||
--- | ||
|
||
# SingleStore Setup Guide {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%} | ||
|
||
Follow our setup guide to connect SingleStore to Fivetran. | ||
|
||
> NOTE: This connector is [partner-built](/docs/partner-built-program). For any questions related | ||
> to SingleStore connector and its documentation, contact SingleStore by raising an issue in the | ||
> [SingleStore Fivetran Connector](https://github.com/singlestore-labs/singlestore-fivetran-connector) | ||
> GitHub repository. | ||
----- | ||
|
||
## Prerequisites | ||
|
||
To connect your SingleStore database to Fivetran, you need: | ||
|
||
- A SingleStore instance of version 8.7.16 or higher. Refer | ||
to [Creating and Using Workspaces](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/creating-and-using-workspaces/) | ||
for instructions on creating a SingleStore workspace in | ||
the [Cloud Portal](https://portal.singlestore.com/). | ||
To deploy a Self-Managed cluster instead, refer | ||
to [Deploy](https://docs.singlestore.com/db/latest/deploy/). Once the SingleStore | ||
workspace/cluster is Active, you'll need the following to connect to Fivetran: | ||
- `Host` | ||
- `Port` | ||
- `Username` | ||
- `Password` | ||
- `Database` | ||
- `Table` | ||
- A Fivetran account with | ||
the [Connector Creator](/docs/using-fivetran/fivetran-dashboard/account-management/role-based-access-control#rbacpermissions) | ||
permissions. | ||
|
||
--- | ||
|
||
## Setup instructions | ||
|
||
To authorize Fivetran to connect to your SinlgeStore database, follow these instructions: | ||
|
||
### <span class="step-item">Configure SingleStore</span> | ||
|
||
1. Configure your firewall and/or other access control systems to allow incoming connections to your | ||
SingleStore instance from [Fivetran's IPs](https://fivetran.com/docs/using-fivetran/ips) for your | ||
region. | ||
2. Ensure that the SingleStore database user has the `SELECT` permission. | ||
3. Enable support of OBSERVE queries | ||
|
||
``` | ||
SET enable_observe_queries=1 | ||
``` | ||
|
||
### <span class="step-item">Finish Fivetran configuration </span> | ||
|
||
1. Log in to your Fivetran account. | ||
2. Go to the [**Connectors** page](https://fivetran.com/dashboard/connectors), and then click * | ||
*+ Add connector**. | ||
3. Select **SingleStore** as the connector type. | ||
4. Enter the following connection configurations for you SingleStore workspace/cluster: | ||
* **Host** | ||
* **Port** | ||
* **Database** | ||
* **Table** | ||
* **Username** | ||
* **Password** | ||
5. (Optional) Enable SSL and specify related configurations. | ||
6. (Optional) Specify additional **Driver Parameters**. Refer | ||
to [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#connection-string-parameters) | ||
documentation for a list of supported parameters. | ||
7. Click **Save & Test**. | ||
|
||
### Setup tests | ||
|
||
Fivetran performs the following SingleStore connection tests: | ||
|
||
- The Connection test checks if Fivetran can connect to your SingleStore cluster using credentials | ||
provided in the setup form | ||
- The Table test checks if specified table exists | ||
|
||
--- | ||
|
||
## Related articles | ||
|
||
[<i aria-hidden="true" class="material-icons">description</i> Connector Overview](/docs/connectors/databases/singlestore) | ||
|
||
<b> </b> | ||
|
||
[<i aria-hidden="true" class="material-icons">account_tree</i> Schema Information](/docs/connectors/databases/singlestore#schemainformation) | ||
|
||
<b> </b> | ||
|
||
[<i aria-hidden="true" class="material-icons">settings</i> API Connector Configuration](/docs/rest-api/connectors/config#sinlgestore) | ||
|
||
<b> </b> | ||
|
||
[<i aria-hidden="true" class="material-icons">home</i> Documentation Home](/docs/getting-started) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
{ | ||
} |