Skip to content

Commit

Permalink
Added README and docs
Browse files Browse the repository at this point in the history
  • Loading branch information
Adalbert Makarovych committed Oct 21, 2024
1 parent 2690fed commit d74a7a5
Show file tree
Hide file tree
Showing 5 changed files with 311 additions and 1 deletion.
97 changes: 97 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# SingleStore Fivetran Connector

## Pre-requisites for development

- JDK v17
- Gradle 8 ([here](https://gradle.org/install/#manually) is an installation instruction)

## Steps for starting server

1. Build the Jar

```
gradle jar
```

2. Run the Jar

```
java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar
```

## Steps for running Java tests

1. Start SingleStore cluster

```
docker run \
-d --name singlestoredb-dev \
-e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \
-e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \
-e SINGLESTORE_VERSION="8.7.16" \
-p 3306:3306 -p 8080:8080 -p 9000:9000 \
ghcr.io/singlestore-labs/singlestoredb-dev:latest
```

2. Create `ROOT_PASSWORD` environment variable

```
export ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD"
```

3. Run tests

```
gradle build
```

## Steps for using Connector tester

1. Start SingleStore cluster

```
docker run \
-d --name singlestoredb-dev \
-e SINGLESTORE_LICENSE="YOUR SINGLESTORE LICENSE" \
-e ROOT_PASSWORD="YOUR SINGLESTORE ROOT PASSWORD" \
-e SINGLESTORE_VERSION="8.7.16" \
-p 3306:3306 -p 8080:8080 -p 9000:9000 \
ghcr.io/singlestore-labs/singlestoredb-dev:latest
```

2. Create database and table

```
DROP DATABASE IF EXISTS tester;
CREATE DATABASE tester;
USE tester;
CREATE TABLE t(a INT PRIMARY KEY, b INT);
```

3. Start Connector server

```
gradle jar
java -jar build/libs/singlestore-fivetran-connector-0.0.1.jar
```

4. Run the tester by following instructions
from [here](https://github.com/fivetran/fivetran_sdk/blob/main/tools/destination-tester/README.md).
As a command use you can use

```
docker run --mount type=bind,source=./tester,target=/data -a STDIN -a STDOUT -a STDERR -it -e GRPC_HOSTNAME=localhost --network=host fivetrandocker/fivetran-sdk-tester:0.24.0729.001 --tester-type source --port 55051
```

5. Update table

```
INSERT INTO t VALUES(1, 2);
INSERT INTO t VALUES(2, 2);
DELETE FROM t WHERE a = 1;
UPDATE t SET b = 3 WHERE a = 2;
```

6. Check the content of `./tester/warehouse.db` file
using [DuckDB](https://duckdb.org/docs/api/cli/overview.html) CLI
or [DBeaver](https://duckdb.org/docs/guides/sql_editors/dbeaver)
110 changes: 110 additions & 0 deletions docs/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
---
name: SingleStore
title: SingleStore connector by Fivetran | Fivetran documentation
Description: Connect your SingleStore data to your destination using Fivetran.
hidden: false
---

# SingleStore {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%}

[SingleStore](https://www.singlestore.com/) is a distributed, cloud-native database that can handle
transactional and analytical workloads with a unified engine. It provides real-time analytics,
transactions, and streaming capabilities, enabling users to handle diverse workloads on a single
platform.

------------------

## Features

{% featureTable connector="singlestore" %}
Capture Deletes:
Custom Data:
{% /featureTable %}

------------------

## Setup guide

Follow our [step-by-step SingleStore setup guide](/docs/{path}/setup-guide) to connect
SingleStore with your destination using Fivetran connectors.

------------------

## Sync overview

Once Fivetran is connected to your SingleStore deployment, the connector fetches an initial
consistent snapshot of all data from your SingleStore table. Once the initial sync is complete, the
connector stream UPDATE/DELETE/INSERT events made to your SingleStore table.

SingleStore connector
uses [OBSERVE](https://docs.singlestore.com/cloud/reference/sql-reference/data-manipulation-language-dml/observe/)
to capture change events.

Connector does not support handling schema changes. You cannot run `ALTER` and `DROP` queries
while the `OBSERVE` query is running.

------------------

## Schema information

Fivetran replicates a single table of the SingleStore database. Selected SingleStore database is
mapped to Fivetran schema.

### Fivetran-generated columns

Fivetran adds the following columns to table in your destination:

- `_fivetran_deleted` (BOOLEAN) marks deleted rows in the source database.
- `_fivetran_synced` (UTC TIMESTAMP) indicates when Fivetran last successfully synced the row.
- `_fivetran_index` (INTEGER) shows the order of updates for tables that do not have a primary key.
- `_fivetran_id` (STRING) is the hash of the non-Fivetran values of each row. It's a unique ID that
Fivetran uses to avoid duplicate rows in tables that do not have a primary key.

### Type transformations and mapping

As we extract your data, we match SingleStore data types in your SingleStore database to types that
Fivetran supports. If we don't support a specific data type, we automatically change that type to
the closest supported type.

The following table illustrates how we transform your SingleStore data types into Fivetran supported
types:

| SingleStore Data Type | Fivetran Data Type | Notes |
|-----------------------|--------------------|------------------------------------------------------------------------------------------------------------------------------------|
| BOOL | SHORT |
| BIT | BINARY |
| TINYINT | SHORT |
| SMALLINT | SHORT |
| MEDIUMINT | INT |
| INT | INT |
| BIGINT | LONG |
| FLOAT | FLOAT |
| DOUBLE | DOUBLE |
| DECIMAL | DECIMAL |
| DATE | NAIVE_DATE |
| TIME | NAIVE_DATETIME |
| TIME(6) | NAIVE_DATETIME |
| DATETIME | NAIVE_DATETIME |
| DATETIME(6) | NAIVE_DATETIME |
| TIMESTAMP | NAIVE_DATETIME |
| TIMESTAMP(6) | NAIVE_DATETIME |
| YEAR | NAIVE_DATE |
| CHAR | STRING |
| VARCHAR | STRING |
| TINYTEXT | STRING |
| TEXT | STRING |
| MEDIUMTEXT | STRING |
| LONGTEXT | STRING |
| BINARY | BINARY |
| VARBINARY | BINARY |
| TINYBLOB | BINARY |
| BLOB | BINARY |
| MEDIUMBLOB | BINARY |
| LONGBLOB | BINARY |
| JSON | JSON |
| BSON | BINARY |
| GEOGRAPHY | STRING |
| GEOGRAPHYPOINT | STRING |
| ENUM | STRING |
| SET | STRING |
| VECTOR | BINARY/JSON | If the `vector_type_project_format` variable is set to `BINARY` - `VECTOR` is mapped to `BINARY`, otherwise it is mapped to `JSON` |
101 changes: 101 additions & 0 deletions docs/setup-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
name: SingleStore
title: SingleStore source connector for Fivetran
description: Read step-by-step instructions on how to connect SingleStore with your destination using Fivetran connectors.
hidden: false
---

# SingleStore Setup Guide {% typeBadge connector="singlestore" /%} {% availabilityBadge connector="singlestore" /%}

Follow our setup guide to connect SingleStore to Fivetran.

> NOTE: This connector is [partner-built](/docs/partner-built-program). For any questions related
> to SingleStore connector and its documentation, contact SingleStore by raising an issue in the
> [SingleStore Fivetran Connector](https://github.com/singlestore-labs/singlestore-fivetran-connector)
> GitHub repository.
-----

## Prerequisites

To connect your SingleStore database to Fivetran, you need:

- A SingleStore instance of version 8.7.16 or higher. Refer
to [Creating and Using Workspaces](https://docs.singlestore.com/cloud/getting-started-with-singlestore-helios/about-workspaces/creating-and-using-workspaces/)
for instructions on creating a SingleStore workspace in
the [Cloud Portal](https://portal.singlestore.com/).
To deploy a Self-Managed cluster instead, refer
to [Deploy](https://docs.singlestore.com/db/latest/deploy/). Once the SingleStore
workspace/cluster is Active, you'll need the following to connect to Fivetran:
- `Host`
- `Port`
- `Username`
- `Password`
- `Database`
- `Table`
- A Fivetran account with
the [Connector Creator](/docs/using-fivetran/fivetran-dashboard/account-management/role-based-access-control#rbacpermissions)
permissions.

---

## Setup instructions

To authorize Fivetran to connect to your SinlgeStore database, follow these instructions:

### <span class="step-item">Configure SingleStore</span>

1. Configure your firewall and/or other access control systems to allow incoming connections to your
SingleStore instance from [Fivetran's IPs](https://fivetran.com/docs/using-fivetran/ips) for your
region.
2. Ensure that the SingleStore database user has the `SELECT` permission.
3. Enable support of OBSERVE queries

```
SET enable_observe_queries=1
```

### <span class="step-item">Finish Fivetran configuration </span>

1. Log in to your Fivetran account.
2. Go to the [**Connectors** page](https://fivetran.com/dashboard/connectors), and then click *
*+ Add connector**.
3. Select **SingleStore** as the connector type.
4. Enter the following connection configurations for you SingleStore workspace/cluster:
* **Host**
* **Port**
* **Database**
* **Table**
* **Username**
* **Password**
5. (Optional) Enable SSL and specify related configurations.
6. (Optional) Specify additional **Driver Parameters**. Refer
to [The SingleStore JDBC Driver](https://docs.singlestore.com/cloud/developer-resources/connect-with-application-development-tools/connect-with-java-jdbc/the-singlestore-jdbc-driver/#connection-string-parameters)
documentation for a list of supported parameters.
7. Click **Save & Test**.

### Setup tests

Fivetran performs the following SingleStore connection tests:

- The Connection test checks if Fivetran can connect to your SingleStore cluster using credentials
provided in the setup form
- The Table test checks if specified table exists

---

## Related articles

[<i aria-hidden="true" class="material-icons">description</i> Connector Overview](/docs/connectors/databases/singlestore)

<b> </b>

[<i aria-hidden="true" class="material-icons">account_tree</i> Schema Information](/docs/connectors/databases/singlestore#schemainformation)

<b> </b>

[<i aria-hidden="true" class="material-icons">settings</i> API Connector Configuration](/docs/rest-api/connectors/config#sinlgestore)

<b> </b>

[<i aria-hidden="true" class="material-icons">home</i> Documentation Home](/docs/getting-started)
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ public void update(UpdateRequest request, StreamObserver<UpdateResponse>

try {
State state;
if (request.hasStateJson()) {
if (request.hasStateJson() && !request.getStateJson().equals("{}")) {
state = State.fromJson(request.getStateJson());
} else {
state = new State(conn.getNumPartitions());
Expand Down
2 changes: 2 additions & 0 deletions tester/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{
}

0 comments on commit d74a7a5

Please sign in to comment.