-
Notifications
You must be signed in to change notification settings - Fork 30
feature(examples): Add ZeroHash connector #447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fivertran-karunveluru
wants to merge
6
commits into
main
Choose a base branch
from
zerohash
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
b365af0
feature(examples): Add ZeroHash connector
fivertran-karunveluru 9a38802
Refactor README.md for better readability
fivetran-dejantucakov 19cacdd
Merge branch 'main' into zerohash
fivertran-karunveluru e2ae51c
feature(examples): Address PR comments
fivertran-karunveluru 599a4b5
Merge branch 'zerohash' of https://github.com/fivetran/fivetran_conne…
fivertran-karunveluru dd4a84c
Update README.md
fivetran-dejantucakov File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| # ZeroHash API Connector Example | ||
|
|
||
| ## Connector overview | ||
| This connector syncs participants, accounts, and assets data from the ZeroHash API. It demonstrates how to implement HMAC-SHA256 authentication, handle cryptocurrency and digital asset data, and process financial information using memory-efficient streaming patterns. The connector fetches participant information, account balances, and supported asset definitions from ZeroHash's certification environment. | ||
|
|
||
| ## Requirements | ||
| - [Supported Python versions](https://github.com/fivetran/fivetran_connector_sdk/blob/main/README.md#requirements) | ||
| - Operating system: | ||
| - Windows: 10 or later (64-bit only) | ||
| - macOS: 13 (Ventura) or later (Apple Silicon [arm64] or Intel [x86_64]) | ||
| - Linux: Distributions such as Ubuntu 20.04 or later, Debian 10 or later, or Amazon Linux 2 or later (arm64 or x86_64) | ||
|
|
||
| ## Getting started | ||
| Refer to the [Connector SDK Setup Guide](https://fivetran.com/docs/connectors/connector-sdk/setup-guide) to get started. | ||
|
|
||
| ## Features | ||
| - Syncs participants, accounts, and assets data from ZeroHash API | ||
| - HMAC-SHA256 authentication with automatic signature generation (refer to the `__generate_signature` function) | ||
| - Memory-efficient streaming prevents data accumulation for large datasets (refer to the `get_participants`, `get_accounts`, and `get_assets` functions) | ||
| - Comprehensive error handling with exponential backoff retry logic (refer to the `__handle_rate_limit` and `__handle_request_error` functions) | ||
| - Incremental synchronization using timestamp-based cursors (refer to the `get_time_range` function) | ||
| - Cryptocurrency and digital asset data processing with proper decimal handling | ||
|
|
||
| ## Configuration file | ||
| ```json | ||
| { | ||
| "api_key": "<YOUR_ZEROHASH_API_KEY>", | ||
| "secret_key": "<YOUR_ZEROHASH_SECRET_KEY>", | ||
| "sync_frequency_hours": "<SYNC_FREQUENCY_HOURS>", | ||
| "initial_sync_days": "<INITIAL_SYNC_DAYS>", | ||
| "request_timeout_seconds": "<REQUEST_TIMEOUT_SECONDS>", | ||
| "retry_attempts": "<RETRY_ATTEMPTS>", | ||
| "enable_incremental_sync": "<ENABLE_INCREMENTAL_SYNC>", | ||
| "enable_debug_logging": "<ENABLE_DEBUG_LOGGING>" | ||
| } | ||
| ``` | ||
|
|
||
| Configuration parameters: | ||
|
|
||
| - `api_key` (required): Your ZeroHash API public key | ||
| - `secret_key` (required): Your ZeroHash API secret key for signing requests | ||
| - `sync_frequency_hours`: How often to sync data in hours | ||
| - `initial_sync_days`: Number of days to sync on first run | ||
| - `request_timeout_seconds`: Timeout for API requests | ||
| - `retry_attempts`: Number of retry attempts for failed requests | ||
| - `enable_incremental_sync`: Enable timestamp-based incremental syncing | ||
| - `enable_debug_logging`: Enable detailed logging for troubleshooting | ||
fivertran-karunveluru marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Requirements file | ||
| This connector does not require any additional packages beyond those provided by the Fivetran environment. | ||
|
|
||
| Note: The `fivetran_connector_sdk:latest` and `requests:latest` packages are pre-installed in the Fivetran environment. To avoid dependency conflicts, do not declare them in your `requirements.txt`. | ||
|
|
||
| ## Authentication | ||
| 1. Log in to the [ZeroHash Developer Portal](https://api.cert.zerohash.com). | ||
| 2. Register a new application to obtain API credentials. | ||
| 3. Make a note of the `api_key` (public key) and `secret_key` from your application settings. | ||
| 4. Retrieve your organization details from ZeroHash administrators. | ||
| 5. Use certification environment credentials for testing, production credentials for live syncing. | ||
|
|
||
| Note: The connector automatically handles HMAC-SHA256 signature generation for each request. Credentials are never logged or exposed in plain text. Each request is signed with a timestamp and request details for security. | ||
|
|
||
| ## Pagination | ||
| ZeroHash API responses are processed as complete datasets without pagination. The connector uses generator-based processing (refer to `get_participants`, `get_accounts`, and `get_assets` functions) to prevent memory accumulation while yielding individual records for immediate processing. | ||
|
|
||
| ## Data handling | ||
| Financial and cryptocurrency data is mapped from ZeroHash's API format to normalized database columns (refer to the `__map_participant_data`, `__map_account_data`, and `__map_asset_data` functions). Account balances and asset amounts are preserved as strings to maintain precision. All timestamps are converted to UTC format for consistency. | ||
|
|
||
| Supports timestamp-based incremental synchronization using the `last_sync_time` state parameter (refer to the `get_time_range` function). Initial sync can be configured to fetch historical data up to 365 days. | ||
|
|
||
| ## Error handling | ||
| - 429 Rate Limited: Automatic retry with exponential backoff (refer to the `__handle_rate_limit` function) | ||
| - Timeout handling with configurable retry attempts (refer to the `__handle_request_error` function) | ||
| - Authentication errors with clear guidance for credential setup | ||
| - Network connectivity issues with exponential backoff and jitter | ||
| - Parameter validation with descriptive error messages provides clear guidance for fixing setup issues | ||
|
|
||
| ## Tables created | ||
| | Table | Primary Key | Description | | ||
| |-------|-------------|-------------| | ||
| | PARTICIPANTS | `id` | Participant information including customers and businesses | | ||
| | ACCOUNTS | `id` | Account balances and asset holdings for each participant | | ||
| | ASSETS | `id` | Supported cryptocurrency and digital asset definitions | | ||
|
|
||
| Column types are automatically inferred by Fivetran. Sample columns include `email`, `participant_code`, `name`, `status`, `account_id`, `asset_symbol`, `balance`, `available_balance`, `symbol`, `decimals`, `minimum_amount`, `maximum_amount`. | ||
fivertran-karunveluru marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Additional considerations | ||
| The examples provided are intended to help you effectively use Fivetran's Connector SDK. While we've tested the code, Fivetran cannot be held responsible for any unexpected or negative consequences that may arise from using these examples. For inquiries, please reach out to our Support team. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "api_key": "<YOUR_ZEROHASH_API_KEY>", | ||
| "secret_key": "<YOUR_ZEROHASH_SECRET_KEY>", | ||
| "sync_frequency_hours": "<SYNC_FREQUENCY_HOURS>", | ||
| "initial_sync_days": "<INITIAL_SYNC_DAYS>", | ||
| "request_timeout_seconds": "<REQUEST_TIMEOUT_SECONDS>", | ||
| "retry_attempts": "<RETRY_ATTEMPTS>", | ||
| "enable_incremental_sync": "<ENABLE_INCREMENTAL_SYNC>", | ||
| "enable_debug_logging": "<ENABLE_DEBUG_LOGGING>" | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.