Load datasets from KYVE pools to BigQuery and Postgres
git clone https://github.com/KYVENetwork/kyve-dlt.git
cd kyve-dlt
make build
This will build dlt in /build directory. Afterward, you may want to put it into your machine's PATH like as follows:
cp build/dlt ~/go/bin/dlt
To set up the dlt
config, run:
dlt init
This will either guide you to set up a first source and destination or create a config with default values. A connection consisting of a source and a destination is required to start any sync process.
Depending on what you want to achieve with dlt
there are two commands available. A quick summary of what they do
and when to use them can be found below:
Description | Recommendation | |
---|---|---|
load | Loads data from a KYVE source into a destination. | Generally recommended to load a dataset into a destination. |
sync | Runs a supervised incremental loading process in cronjob to keep the data destination in sync with the data source. | Recommended to keep an already loaded dataset updated with incrementally added data. |
Usage:
dlt load --connection connection_1
To start the loading process from or to a certain bundle, simply use the --from-bundle-id
or --to-bundle-id
flags.
dlt
always checks if a bundle was already loaded into a destination. By default, the to-bundle-id
is set to the latest found bundle ID + 1
.
To force the loading of the specified range of bundle, simple use the --force
flag.
Usage:
dlt sync --connection connection_1
sync
executes the loading process in a cronjob. Therefore, specify the connection -> cron
field in the config.
The following cron syntax is expected: * * * * *
. The supervising process always checks if an underlying loading process
is currently running to prevent parallelized syncs. It waits until all processes are finished to proceed with the sync.
With the following commands, sources, destinations, and connections can be added, removed or listed:
dlt sources {add|remove|list}
dlt destinations {add|remove|list}
dlt connections {add|remove|list}
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"_dlt_raw_id": { "type": "string" },
"_dlt_extracted_at": { "type": "string" },
"key": { "type": "string" },
"value": { "type": "string" },
"bundle_id": { "type": "integer" }
},
"required": ["_dlt_raw_id", "_dlt_extracted_at", "key", "value", "bundle_id"]
}
This schema supports all KYVE datasets by default and consists of the core key
and value
structure that is required by the KYVE protocol.
The key
is the unique identifier of the data item in a data pool (e.g. height of a block, timestamp), whereas the value
includes the actual data.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"_dlt_raw_id": { "type": "string" },
"_dlt_extracted_at": { "type": "string" },
"height": { "type": "integer" },
"value": { "type": "string" },
"bundle_id": { "type": "integer" }
},
"required": ["_dlt_raw_id", "_dlt_extracted_at", "key", "value", "bundle_id"]
}
This schema is supported for all KYVE pools with an INTEGER as key (runtime: @kyvejs/tendermint
, @kyvejs/tendermint-bsync
, @kyvejs/evm
). Instead of using the raw key, it converts it to
a height
as integer
.
This schema is supported for all Tendermint pools (runtime: @kyvejs/tendermint
) and preprocesses the events in individual rows.
It is recommended to use for datasets with big data items (e.g. Osmosis). This is the schema:
{
"type": "object",
"properties": {
"_dlt_raw_id": { "type": "string" },
"_dlt_extracted_at": { "type": "string" },
"type": { "type": "string" },
"value": { "type": "string" },
"height": { "type": "string" },
"array_index": { "type": "integer" },
"bundle_id": { "type": "integer" }
},
"required": ["_dlt_raw_id", "_dlt_extracted_at", "type", "value", "height", "array_index", "bundle_id"]
}
For this schema, the height
itself is no unique identifier, because more than one rows are written for a single data item.
The first row includes the block without events in the value
field (type = "block"
). The events with type
begin_block_event
, tx_result
, and end_block_event
follow, including the event value in value
and an array_index
.
This structure allows everyone to reconstruct the data completely.
- BigQuery
- Postgres