From fc359e763999ea10af39ef44a8781d7f09e73231 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Lu=C3=ADs=20Freitas?= Date: Fri, 4 Aug 2023 16:03:34 +0100 Subject: [PATCH 1/5] Update deploy-contract.md (#1460) --- docs/sdk/js/promises/deploy-contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sdk/js/promises/deploy-contract.md b/docs/sdk/js/promises/deploy-contract.md index cbf375e4820..6fbff356f3b 100644 --- a/docs/sdk/js/promises/deploy-contract.md +++ b/docs/sdk/js/promises/deploy-contract.md @@ -6,7 +6,7 @@ sidebar_position: 4 You might want your smart contract to deploy subsequent smart contract code for a few reasons: -- The contract acts as a Factory, a pattern where a parent contract creates many child contracts ([Mintbase](https://www.mintbase.io/) does this to create a new NFT store for [anyone who wants one](https://docs.mintbase.io/creating/store/deploy-fee); [Rainbow Bridge](https://near.org/bridge/) does this to deploy separate Fungible Token contracts for [each bridged token](https://github.com/aurora-is-near/rainbow-token-connector/blob/ce7640da144f000e0a93b6d9373bbc2514e37f3b/bridge-token-factory/src/lib.rs#L311-L341)) +- The contract acts as a Factory, a pattern where a parent contract creates many child contracts ([Mintbase](https://www.mintbase.xyz/) does this to create a new NFT store for [anyone who wants one](https://docs.mintbase.xyz/creating/store/deploy-fee); [Rainbow Bridge](https://near.org/bridge/) does this to deploy separate Fungible Token contracts for [each bridged token](https://github.com/aurora-is-near/rainbow-token-connector/blob/ce7640da144f000e0a93b6d9373bbc2514e37f3b/bridge-token-factory/src/lib.rs#L311-L341)) - The contract [updates its own code](../../../2.develop/upgrade.md#programmatic-update) (calls `deploy` on itself). - You could implement a "contract per user" system that creates app-specific subaccounts for users (`your-app.user1.near`, `your-app.user2.near`, etc) and deploys the same contract to each. This is currently prohibitively expensive due to NEAR's [storage fees](https://docs.near.org/concepts/storage/storage-staking), but that may be optimized in the future. If it is, this sort of "sharded app design" may become the more scalable, user-centric approach to contract standards and app mechanics. An early experiment with this paradigm was called [Meta NEAR](https://github.com/metanear). From 2ef2ef0e7af80bec76ca9670f9dbe61a91f5989c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dami=C3=A1n=20Parrino?= Date: Tue, 8 Aug 2023 16:45:32 -0300 Subject: [PATCH 2/5] Update indexer4explorer.md (#1461) --- docs/4.tools/indexer4explorer.md | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/docs/4.tools/indexer4explorer.md b/docs/4.tools/indexer4explorer.md index 477b0572b31..1bacf0838e0 100644 --- a/docs/4.tools/indexer4explorer.md +++ b/docs/4.tools/indexer4explorer.md @@ -27,7 +27,7 @@ You can find the source code on [this GitHub repository](https://github.com/near :::warning The database could go down or take time to reflect the most current blockchain info. Do not use it in production. -For a reliable source of real time and past information, please run your own [indexer](https://near-indexers.io/). +For a reliable source of real-time and past information, please run your own [indexer](https://near-indexers.io/). ::: --- @@ -56,19 +56,11 @@ You can use any database manager compatible with PostgreSQL. If you don't know a - ---- - -## Database Structure -Please bear in mind that the structure might evolve or change with time. - -[![structure_img](https://raw.githubusercontent.com/near/near-indexer-for-explorer/master/docs/near-indexer-for-explorer-db.png)](https://raw.githubusercontent.com/near/near-indexer-for-explorer/master/docs/near-indexer-for-explorer-db.png) - --- ## Example Queries ### Transactions Calling a Method -Query for all transaction that called `contribute` in the `v1.faucet.nonofficial.testnet` testnet account. +Query for all transactions that called `contribute` in the `v1.faucet.nonofficial.testnet` testnet account. ```sql select r.predecessor_account_id, t.transaction_hash @@ -82,7 +74,7 @@ where r.receiver_account_id ='v1.faucet.nonofficial.testnet'
-### Users, Status and Attached Money +### Users, Status, and Attached Money Query for all users that called `contribute` in `v1.faucet.nonofficial.testnet`, how much they attached to the call, and the transaction status. ```sql @@ -105,4 +97,4 @@ select r.receiver_account_id, ara.args -> 'deposit' as deposit from receipts r, action_receipt_actions ara where r.predecessor_account_id ='v1.faucet.nonofficial.testnet' and ara.receipt_id = r.receipt_id and ara.action_kind = 'TRANSFER' -``` \ No newline at end of file +``` From e7350a913e3e5189c10c3c64160a456421cedc03 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dami=C3=A1n=20Parrino?= Date: Wed, 9 Aug 2023 15:15:39 -0300 Subject: [PATCH 3/5] run near lake (#1463) --- docs/3.tutorials/indexer/run-near-lake.md | 226 ++++++++++++++++++++++ website/sidebars.json | 1 + 2 files changed, 227 insertions(+) create mode 100644 docs/3.tutorials/indexer/run-near-lake.md diff --git a/docs/3.tutorials/indexer/run-near-lake.md b/docs/3.tutorials/indexer/run-near-lake.md new file mode 100644 index 00000000000..92aa0837bf9 --- /dev/null +++ b/docs/3.tutorials/indexer/run-near-lake.md @@ -0,0 +1,226 @@ +--- +id: run-lake-indexer +sidebar_label: Running Lake Indexer +--- + +# Running NEAR Lake Indexer + +:::info + +NEAR Lake is a blockchain indexer built on top of [NEAR Indexer microframework](https://github.com/nearprotocol/nearcore/tree/master/chain/indexer) +to watch the network and store all the events as JSON files on AWS S3. + +::: + +## How to start + +The Lake Indexer setup consists of the following components: +* AWS S3 Bucket as a storage +* NEAR Lake binary that operates as a regular NEAR Protocol peer-to-peer node, so you will operate it as + any other [Regular/RPC Node in NEAR](https://near-nodes.io/rpc/hardware-rpc) + +### Prepare Development Environment + +Before you proceed, make sure you have the following software installed: +* [Rust compiler](https://rustup.rs/) of the version that is mentioned in `rust-toolchain` file in the root of + [nearcore](https://github.com/nearprotocol/nearcore) project. +* Ensure you have [AWS Credentials configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) + From AWS Docs: + + > For example, the files generated by the AWS CLI for a default profile configured with aws configure looks similar to the following. + > + > ~/.aws/credentials + > ``` + > [default] + > aws_access_key_id=AKIAIOSFODNN7EXAMPLE + > aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + > ``` + +### Compile NEAR Lake + +```bash +$ cargo build --release +``` + +### Configure NEAR Lake + +To connect NEAR Lake to the specific chain you need to have necessary configs, you can generate it as follows: + +```bash +$ ./target/release/near-lake --home ~/.near/testnet init --chain-id testnet --download-config --download-genesis +``` + +The above code will download the official genesis config and generate necessary configs. You can replace `testnet` in the command above to different network ID (`betanet`, `mainnet`). + +:::info nearcore configuration + +According to changes in `nearcore` config generation we don't fill all the necessary fields in the config file. +While [this issue is open](https://github.com/nearprotocol/nearcore/issues/3156) you need to download config you want and replace the generated one manually. +- [testnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/config.json) +- [betanet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/betanet/config.json) +- [mainnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/config.json) + +::: + +Configs for the specified network are in the `--home` provided folder. We need to ensure that NEAR Lake follows +all the necessary shards, so `"tracked_shards"` parameters in `~/.near/testnet/config.json` needs to be configured properly. +Currently, `nearcore` treats empty value for `"tracked_shards"` as "do not track any shard" and **any value** as "track all shards". +For example, in order to track all shards, you just add the shard #0 to the list: + +``` +... +"tracked_shards": [0], +... +``` + +### Run NEAR Lake + +Commands to run NEAR Lake, after `./target/release/near-lake` + +| Command | Key/Subcommand | Required/Default | Responsible for | +|--------- |-------------------------- |------------------------------------------------------------------ |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| | `--home` | Default
`~/.near` | Tells the node where too look for necessary files:
`config.json`
,
`genesis.json`
,
`node_key.json`
, and
`data`
folder | +| `init` | | | Tells the node to generate config files in `--home-dir` | +| | `--chain-id` | Required

* `localnet`
* `testnet`
* `mainnet` | Defines the chain to generate config files for | +| | `--download-config` | Optional | If provided tells the node to download `config.json` from the public URL. You can download them manually

- [testnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/config.json)
- [mainnet config.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/config.json) | +| | `--download-genesis` | Optional | If provided tells the node to download `genesis.json` from the public URL. You can download them manually

- [testnet genesis.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/testnet/genesis.json)
- [mainnet genesis.json](https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/mainnet/genesis.json) | +| | TODO:
Other `neard` keys | | | +| `run` | | | Runs the node | +| | `--bucket` | Required | AWS S3 Bucket name | +| | `--region` | Required | AWS S3 Bucket region | +| | `--fallback-region` | Default eu-central-1 | AWS S3 Fallback region | +| | `--endpoint` | Optional | AWS S3 compatible API endpoint | +| | `--stream-while-syncing` | Optional | If provided Indexer streams blocks while they appear on the node instead of waiting the node to be fully synced | +| | `--concurrency` | Default 1 | Defines the concurrency for the process of saving block data to AWS S3 | +| | `sync-from-latest` | One of the `sync-` subcommands is required | Tells the node to start indexing from the latest block in the network | +| | `sync-from-interruption` | One of the `sync-` subcommands is required | Tells the node to start indexing from the block the node was interrupted on (if it is a first start it will fallback to `sync-from-latest`) | +| | `sync-from-block --height N` | One of the
`sync-`
subcommands is required | Tells the node to start indexing from the specified block height `N` (**Ensure** you node data has the block you want to start from) | + +```bash +$ ./target/release/near-lake --home ~/.near/testnet run --stream-while-syncing --concurrency 50 sync-from-latest +``` + +After the network is synced, you should see logs of every block height currently received by NEAR Lake. + + +## Syncing + +Whenever you run NEAR Lake for any network except localnet you'll need to sync with the network. +This is required because it's a natural behavior of `nearcore` node and NEAR Lake is a wrapper +for the regular `nearcore` node. In order to work and index the data your node must be synced +with the network. This process can take a while, so we suggest to download a fresh backup of +the `data` folder and put it in you `--home-dir` of your choice (by default it is `~/.near`) + +:::tip +Running your NEAR Lake node on top of a backup data will reduce the time of syncing process +because your node will download only the data after the backup was cut and it takes reasonable amount time. +::: + +All the backups can be downloaded from the public S3 bucket which contains latest daily snapshots: + +You will need [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to be installed in order to download the backups. + +### Mainnet + +``` +$ aws s3 --no-sign-request cp s3://near-protocol-public/backups/mainnet/rpc/latest . +$ LATEST=$(cat latest) +$ aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/mainnet/rpc/$LATEST ~/.near/data +``` + +### Testnet + +``` +$ aws s3 --no-sign-request cp s3://near-protocol-public/backups/testnet/rpc/latest . +$ LATEST=$(cat latest) +$ aws s3 --no-sign-request cp --no-sign-request --recursive s3://near-protocol-public/backups/testnet/rpc/$LATEST ~/.near/data +``` + + + +## Running NEAR Lake as an archival node + +It's not necessary but in order to index everything in the network it is better to do it from the genesis. +`nearcore` node is running in non-archival mode by default. That means that the node keeps data only +for [5 last epochs](https://docs.near.org/concepts/basics/epoch). In order to index data from the genesis +you need to turn the node in archival mode. + +To do it you need to update `config.json` located in `--home-dir` (by default it is `~/.near`). + +Find next keys in the config and update them as following: + +```json +{ + ... + "archive": true, + "tracked_shards": [0], + ... +} +``` + +The syncing process in archival mode can take a lot of time, so it's better to download a backup provided by NEAR +and put it in your `data` folder. After that your node will download only the data after the backup was cut and it +takes reasonable amount time. + + +All the backups can be downloaded from the public S3 bucket which contains the latest daily snapshots: + +* [Archival Mainnet data folder](https://near-protocol-public.s3-accelerate.amazonaws.com/backups/mainnet/archive/data.tar) +* [Archival Testnet data folder](https://near-protocol-public.s3-accelerate.amazonaws.com/backups/testnet/archive/data.tar) + +See [this link](https://near-nodes.io/archival/run-archival-node-with-nearup) for reference + +## Using the data + +We write all the data to AWS S3 buckets: + +- `near-lake-data-testnet` (`eu-central-1` region) for testnet +- `near-lake-data-mainnet` (`eu-central-1` region) for mainnet + +## Custom S3 storage + +In case you want to run you own `near-lake` instance and store data in some S3 compatible storage ([Minio](https://min.io/) or [Localstack](https://localstack.cloud/) as example) +You can override default S3 API endpoint by using `--endpoint` option + +- run `minio` + +```bash +$ mkdir -p /data/near-lake-custom && minio server /data +``` + +- run `near-lake` + +```bash +$ ./target/release/near-lake --home ~/.near/testnet run --endpoint http://127.0.0.1:9000 --bucket near-lake-custom sync-from-latest +``` + +### Data structure + +The data structure we use is the following: + +``` +/ + block.json + shard_0.json + shard_1.json + ... + shard_N.json +``` + +- `` is a 12-character-long `u64` string with leading zeros (e.g `000042839521`). [See this issue for a reasoning](https://github.com/near/near-lake/issues/23) +- `block_json` contains JSON-serialized [`BlockView`](https://github.com/near/nearcore/blob/e9a28c46c2bea505b817abf484e6015a61ea7d01/core/primitives/src/views.rs#L711-L716) struct. **Note:** this struct might change in the future, we will announce it +- `shard_N.json` where `N` is `u64` starting from `0`. Represents the index number of the shard. In order to find out the expected number of shards in the block you can look in `block.json` at `.header.chunks_included` + +### Access the data + +All NEAR Lake AWS S3 buckets have [Request Payer](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) enabled. It means that anyone with their own AWS credentials can List and Read the bucket's content and **be charged for it by AWS**. Connections to the bucket have to be done with AWS credentials provided. See [NEAR Lake Framework](https://github.com/near/near-lake-framework) for a reference. + +### NEAR Lake Framework + +Once we [set up the public access to the buckets](https://github.com/near/near-lake/issues/22) anyone will be able to build their own code to read it through. + +For our own needs we are working on [NEAR Lake Framework](https://github.com/near/near-lake-framework) to have a simple way to create an indexer on top of the data stored by NEAR Lake itself. + +:::note +See the official NEAR Lake Framework [announcement on the NEAR Gov Forum](https://gov.near.org/t/announcement-near-lake-framework-brand-new-word-in-indexer-building-approach/17668). +::: diff --git a/website/sidebars.json b/website/sidebars.json index 6d5b02dedaa..f119f992ca3 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -365,6 +365,7 @@ { "NEAR Lake Indexer": [ "tutorials/indexer/near-lake-state-changes-indexer", + "tutorials/indexer/run-lake-indexer", "tutorials/indexer/js-lake-indexer", "tutorials/indexer/lake-start-options", "tutorials/indexer/python-lake-indexer", From 3026ac36374faaa045035b7e475b7c0c637a6389 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dami=C3=A1n=20Parrino?= Date: Thu, 10 Aug 2023 11:56:23 -0300 Subject: [PATCH 4/5] Lake indexer structs (#1462) * add lake structs * update * adjust section --- docs/2.develop/lake/structures/block.mdx | 140 +++++++++++++++ docs/2.develop/lake/structures/chunk.mdx | 109 ++++++++++++ .../lake/structures/execution_outcome.mdx | 134 +++++++++++++++ docs/2.develop/lake/structures/receipt.mdx | 101 +++++++++++ docs/2.develop/lake/structures/shard.mdx | 47 +++++ .../lake/structures/state_change.mdx | 104 +++++++++++ docs/2.develop/lake/structures/toc.mdx | 27 +++ .../2.develop/lake/structures/transaction.mdx | 161 ++++++++++++++++++ website/sidebars.json | 18 +- 9 files changed, 840 insertions(+), 1 deletion(-) create mode 100644 docs/2.develop/lake/structures/block.mdx create mode 100644 docs/2.develop/lake/structures/chunk.mdx create mode 100644 docs/2.develop/lake/structures/execution_outcome.mdx create mode 100644 docs/2.develop/lake/structures/receipt.mdx create mode 100644 docs/2.develop/lake/structures/shard.mdx create mode 100644 docs/2.develop/lake/structures/state_change.mdx create mode 100644 docs/2.develop/lake/structures/toc.mdx create mode 100644 docs/2.develop/lake/structures/transaction.mdx diff --git a/docs/2.develop/lake/structures/block.mdx b/docs/2.develop/lake/structures/block.mdx new file mode 100644 index 00000000000..9d655f22e1b --- /dev/null +++ b/docs/2.develop/lake/structures/block.mdx @@ -0,0 +1,140 @@ +--- +sidebar_position: 2 +sidebar_label: Block +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import CodeBlock from '@theme/CodeBlock'; + +# `Block` Structure + +## Definition + +`Block` is the main entity in NEAR Protocol blockchain. `Blocks` are produced in NEAR Protocol every second. + +It contains the info: +- about the `Block` producer (`AccountId` of the validator responsible for particular `Block` production) +- Block Header +- List of Chunk Headers + +## Representation + +### `Block` + + + + +```rust links=1 +pub struct BlockView { + pub author: AccountId, + pub header: BlockHeaderView, + pub chunks: Vec, +} +``` + + + + +```ts links=1 +export interface Block { + author: string; + header: BlockHeader; + chunks: ChunkHeader[]; +}; +``` + + + + + +### `BlockHeaderView` + +`BlockHeaderView` contains main info about the block. + + + + +```rust +pub struct BlockHeaderView { + pub height: BlockHeight, + pub prev_height: Option, + pub epoch_id: CryptoHash, + pub next_epoch_id: CryptoHash, + pub hash: CryptoHash, + pub prev_hash: CryptoHash, + pub prev_state_root: CryptoHash, + pub chunk_receipts_root: CryptoHash, + pub chunk_headers_root: CryptoHash, + pub chunk_tx_root: CryptoHash, + pub outcome_root: CryptoHash, + pub chunks_included: u64, + pub challenges_root: CryptoHash, + /// Legacy json number. Should not be used. + pub timestamp: u64, + pub timestamp_nanosec: u64, + pub random_value: CryptoHash, + pub validator_proposals: Vec, + pub chunk_mask: Vec, + pub gas_price: Balance, + pub block_ordinal: Option, + /// TODO(2271): deprecated. + pub rent_paid: Balance, + /// TODO(2271): deprecated. + pub validator_reward: Balance, + pub total_supply: Balance, + pub challenges_result: ChallengesResult, + pub last_final_block: CryptoHash, + pub last_ds_final_block: CryptoHash, + pub next_bp_hash: CryptoHash, + pub block_merkle_root: CryptoHash, + pub epoch_sync_data_hash: Option, + pub approvals: Vec>, + pub signature: Signature, + pub latest_protocol_version: ProtocolVersion, +} +``` + + + + + +```ts links=1 +export interface BlockHeader { + approvals: (string | null)[]; + blockMerkleRoot: string; + blockOrdinal: number; + challengesResult: ChallengeResult[]; + challengesRoot: string; + chunkHeadersRoot: string; + chunkMask: boolean[]; + chunkReceiptsRoot: string; + chunkTxRoot: string; + chunksIncluded: number; + epochId: string; + epochSyncDataHash: string | null; + gasPrice: string; + hash: string; + height: number; + lastDsFinalBlock: string; + lastFinalBlock: string; + latestProtocolVersion: number; + nextBpHash: string; + nextEpochId: string; + outcomeRoot: string; + prevHash: string; + prevHeight: number; + prevStateRoot: string; + randomValue: string; + rentPaid: string; + signature: string; + timestamp: number; + timestampNanosec: string; + totalSupply: string; + validatorProposals: []; + validatorReward: string; +}; +``` + + + diff --git a/docs/2.develop/lake/structures/chunk.mdx b/docs/2.develop/lake/structures/chunk.mdx new file mode 100644 index 00000000000..c569d4fbb0f --- /dev/null +++ b/docs/2.develop/lake/structures/chunk.mdx @@ -0,0 +1,109 @@ +--- +sidebar_position: 3 +sidebar_label: Chunk +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `Chunk` Structure + +## Definition + +`Chunk` of a [`Block`](./block.mdx) is a part of a [`Block`](./block.mdx) from a [Shard](./shard.mdx). The collection of Chunks of the Block forms the NEAR Protocol [`Block`](./block.mdx) + +Chunk contains all the structures that make the Block: +- [Transactions](./transaction.mdx) +- [Receipts](./receipt.mdx) +- [ChunkHeader](#chunkheaderview) + +## `IndexerChunkView` + + + + +```rust links=1 +pub struct ChunkView { + pub author: AccountId, + pub header: ChunkHeaderView, + pub transactions: Vec, + pub receipts: Vec, +} +``` + + + + +```ts links=1 +export interface Chunk { + author: string; + header: ChunkHeader; + transactions: IndexerTransactionWithOutcome[]; + receipts: Receipt[], +} +``` + + + + +## `ChunkHeaderView` + + + + +```rust links=1 +pub struct ChunkHeaderView { + pub chunk_hash: CryptoHash, + pub prev_block_hash: CryptoHash, + pub outcome_root: CryptoHash, + pub prev_state_root: StateRoot, + pub encoded_merkle_root: CryptoHash, + pub encoded_length: u64, + pub height_created: BlockHeight, + pub height_included: BlockHeight, + pub shard_id: ShardId, + pub gas_used: Gas, + pub gas_limit: Gas, + /// TODO(2271): deprecated. + #[serde(with = "u128_dec_format")] + pub rent_paid: Balance, + /// TODO(2271): deprecated. + #[serde(with = "u128_dec_format")] + pub validator_reward: Balance, + #[serde(with = "u128_dec_format")] + pub balance_burnt: Balance, + pub outgoing_receipts_root: CryptoHash, + pub tx_root: CryptoHash, + pub validator_proposals: Vec, + pub signature: Signature, +} +``` + + + + +```ts links=1 +export interface ChunkHeader { + balanceBurnt: number; + chunkHash: string; + encodedLength: number; + encodedMerkleRoot: string; + gasLimit: number; + gasUsed: number; + heightCreated: number; + heightIncluded: number; + outcomeRoot: string; + outgoingReceiptsRoot: string; + prevBlockHash: string; + prevStateRoot: string; + rentPaid: string; + shardId: number; + signature: string; + txRoot: string; + validatorProposals: ValidatorProposal[]; + validatorReward: string; +}; +``` + + + diff --git a/docs/2.develop/lake/structures/execution_outcome.mdx b/docs/2.develop/lake/structures/execution_outcome.mdx new file mode 100644 index 00000000000..8af9b9b1a6f --- /dev/null +++ b/docs/2.develop/lake/structures/execution_outcome.mdx @@ -0,0 +1,134 @@ +--- +id: execution-outcome +sidebar_position: 6 +sidebar_label: ExecutionOutcome +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `ExecutionOutcome` Structure + +## Definition + +ExecutionOutcome is the result of execution of [Transaction](./transaction.mdx) or [Receipt](./receipt.mdx) + +:::info Transaction's ExecutionOutcome + +In the result of the Transaction execution will always be a Receipt. + +::: + +## `ExecutionOutcomeWithIdView` + + + + +```rust links=1 +pub struct ExecutionOutcomeWithIdView { + pub proof: MerklePath, + pub block_hash: CryptoHash, + pub id: CryptoHash, + pub outcome: ExecutionOutcomeView, +} +``` + + + + +```ts links=1 +export type ExecutionOutcomeWithReceipt = { + executionOutcome: { + blockHash: string; + id: string; + outcome: { + executorId: string, + gasBurnt: number, + logs: string[], + metadata: { + gasProfile: string | null; + version: number; + }; + receiptIds: string[], + status: ExecutionStatus, + tokensBurnt: string + }; + proof: ExecutionProof[]; + }; + receipt: Receipt | null; +}; +``` + + + + +## `ExecutionOutcomeView` + + + + +```rust links=1 +pub struct ExecutionOutcomeView { + /// Logs from this transaction or receipt. + pub logs: Vec, + /// Receipt IDs generated by this transaction or receipt. + pub receipt_ids: Vec, + /// The amount of the gas burnt by the given transaction or receipt. + pub gas_burnt: Gas, + /// The amount of tokens burnt corresponding to the burnt gas amount. + /// This value doesn't always equal to the `gas_burnt` multiplied by the gas price, because + /// the prepaid gas price might be lower than the actual gas price and it creates a deficit. + #[serde(with = "u128_dec_format")] + pub tokens_burnt: Balance, + /// The id of the account on which the execution happens. For transaction this is signer_id, + /// for receipt this is receiver_id. + pub executor_id: AccountId, + /// Execution status. Contains the result in case of successful execution. + pub status: ExecutionStatusView, + /// Execution metadata, versioned + #[serde(default)] + pub metadata: ExecutionMetadataView, +} +``` + + + + + [ExecutionOutcomeWithIdView TypeScript](#executionoutcomewithidview) defines this structure already an nested + + + + + +## `ExecutionStatusView` + + + + +```rust links=1 +pub enum ExecutionStatusView { + /// The execution is pending or unknown. + Unknown, + /// The execution has failed. + Failure(TxExecutionError), + /// The final action succeeded and returned some value or an empty vec encoded in base64. + SuccessValue(String), + /// The final action of the receipt returned a promise or the signed transaction was converted + /// to a receipt. Contains the receipt_id of the generated receipt. + SuccessReceiptId(CryptoHash), +} +``` + + + + +```ts links=1 +export type ExecutionStatus = + | { Unknown: unknown } + | { Failure: unknown } + | { SuccessValue: string } + | { SuccessReceiptId: string }; +``` + + + diff --git a/docs/2.develop/lake/structures/receipt.mdx b/docs/2.develop/lake/structures/receipt.mdx new file mode 100644 index 00000000000..161dbf0c41e --- /dev/null +++ b/docs/2.develop/lake/structures/receipt.mdx @@ -0,0 +1,101 @@ +--- +sidebar_position: 7 +sidebar_label: Receipt +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `Receipt` Structure + +## Definition + +Receipt is an internal transaction in NEAR Protocol. + +## `ReceiptView` + + + + +```rust links=1 +pub struct ReceiptView { + pub predecessor_id: AccountId, + pub receiver_id: AccountId, + pub receipt_id: CryptoHash, + + pub receipt: ReceiptEnumView, +} +``` + + + + +```ts links=1 +export type Receipt = { + predecessorId: string; + receipt: ReceiptEnum; + receiptId: string; + receiverId: string; +} +``` + + + + +## `ReceiptEnumView` + + + + +```rust links=1 +pub enum ReceiptEnumView { + Action { + signer_id: AccountId, + signer_public_key: PublicKey, + #[serde(with = "u128_dec_format")] + gas_price: Balance, + output_data_receivers: Vec, + input_data_ids: Vec, + actions: Vec, + }, + Data { + data_id: CryptoHash, + #[serde(with = "option_base64_format")] + data: Option>, + }, +} +``` + + + + +```ts links=1 +export type ReceiptEnum = + | { + Action: { + actions: Action[]; + gasPrice: string; + inputDataIds: string[]; + outputDataReceivers: string[]; + signerId: string; + signerPublicKey: string; + }; + } + | { + Data: { + data: string; + dataId: string; + }; + }; +``` + + + + +## ActionReceipt + +ActionReceipts hold info about what actions to perform. In the cross-contract calls some actions might happen only when specific data is available, such data is in the [DataReceipts](#datareceipt). + +## DataReceipt + +DataReceipts hold the data that is produced my `Promise` during the cross-contract calls. diff --git a/docs/2.develop/lake/structures/shard.mdx b/docs/2.develop/lake/structures/shard.mdx new file mode 100644 index 00000000000..4b6223854b1 --- /dev/null +++ b/docs/2.develop/lake/structures/shard.mdx @@ -0,0 +1,47 @@ +--- +sidebar_position: 4 +sidebar_label: Shard +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `Shard` structure + +## Definition + +`IndexerShard` struct is ephemeral structure, there is no such entity in `nearcore`. We've introduces it as a container in [`near-indexer-primitives`](https://crates.io/crates/near-indexer-primitives). This container includes: + +- shard ID +- [Chunk](./chunk.mdx) that might be absent +- [ExecutionOutcomes](./execution_outcome.mdx) for [Receipts](./receipt.mdx) (these belong to a Shard not to a [Chunk](./chunk.mdx) or a [Block](./block.mdx)) +- [StateChanges](./state_change.mdx) for a Shard + +## `IndexerShard` + + + + +```rust links=1 +pub struct IndexerShard { + pub shard_id: types::ShardId, + pub chunk: Option, + pub receipt_execution_outcomes: Vec, + pub state_changes: views::StateChangesView, +} +``` + + + + +```ts links=1 +export interface Shard { + shard_id: number; + chunk?: Chunk; + receiptExecutionOutcomes: ExecutionOutcomeWithReceipt[]; + stateChanges: StateChange[]; +}; +``` + + + diff --git a/docs/2.develop/lake/structures/state_change.mdx b/docs/2.develop/lake/structures/state_change.mdx new file mode 100644 index 00000000000..1af0fdcb5d9 --- /dev/null +++ b/docs/2.develop/lake/structures/state_change.mdx @@ -0,0 +1,104 @@ +--- +id: state-change +sidebar_position: 9 +sidebar_label: StateChange +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `StateChange` Structure + +## Definition + +This entitiy from `nearcore` describes how account's state has changed and the reason + +## `StateChangeWithCauseView` + + + + +```rust links=1 +pub struct StateChangeWithCauseView { + pub cause: StateChangeCauseView, + #[serde(flatten)] + pub value: StateChangeValueView, +} + +pub enum StateChangeCauseView { + NotWritableToDisk, + InitialState, + TransactionProcessing { tx_hash: CryptoHash }, + ActionReceiptProcessingStarted { receipt_hash: CryptoHash }, + ActionReceiptGasReward { receipt_hash: CryptoHash }, + ReceiptProcessing { receipt_hash: CryptoHash }, + PostponedReceipt { receipt_hash: CryptoHash }, + UpdatedDelayedReceipts, + ValidatorAccountsUpdate, + Migration, + Resharding, +} + +pub enum StateChangeValueView { + AccountUpdate { + account_id: AccountId, + #[serde(flatten)] + account: AccountView, + }, + AccountDeletion { + account_id: AccountId, + }, + AccessKeyUpdate { + account_id: AccountId, + public_key: PublicKey, + access_key: AccessKeyView, + }, + AccessKeyDeletion { + account_id: AccountId, + public_key: PublicKey, + }, + DataUpdate { + account_id: AccountId, + #[serde(rename = "key_base64", with = "base64_format")] + key: StoreKey, + #[serde(rename = "value_base64", with = "base64_format")] + value: StoreValue, + }, + DataDeletion { + account_id: AccountId, + #[serde(rename = "key_base64", with = "base64_format")] + key: StoreKey, + }, + ContractCodeUpdate { + account_id: AccountId, + #[serde(rename = "code_base64", with = "base64_format")] + code: Vec, + }, + ContractCodeDeletion { + account_id: AccountId, + }, +} +``` + + + + + +```ts links=1 +export type StateChange = { + cause: { + receiptHash: string; + type: string; + }; + change: { + accountId: string; + keyBase64: string; + valueBase64: string; + }, + type: string; +}; +``` + + + + diff --git a/docs/2.develop/lake/structures/toc.mdx b/docs/2.develop/lake/structures/toc.mdx new file mode 100644 index 00000000000..22b2fd9c725 --- /dev/null +++ b/docs/2.develop/lake/structures/toc.mdx @@ -0,0 +1,27 @@ +--- +sidebar_position: 1 +sidebar_label: Table of contents +--- + +# Structures. Table of Contents + +The structures described in this section of the docs are indexer-related more than `nearcore` related. Some of the are artificial and are not a part of `nearcore` + +:::note How data flows in NEAR Protocol + +Don't skip the article about the [NEAR Data Flow](https://docs.near.org/concepts/data-flow/near-data-flow) + +::: + +**The main goal of this section is to describe `StreamerMessage` structure from [`near-indexer-primitives`](https://crates.io/crates/near-indexer-primitives)** + +- `StreamerMessage` + - [Block](./block.mdx) + - [BlockHeader](./block.mdx#blockheaderview) + - [Shards](./shard.mdx) + - [Chunk](./chunk.mdx) + - [ChunkHeader](./chunk.mdx#chunkheaderview) + - [Transactions](./transaction.mdx) + - [Receipts](./receipt.mdx) + - [ExecutionOutcomes](./execution_outcome.mdx) for [Receipts](./receipt.mdx) + - [StateChanges](./state_change.mdx) diff --git a/docs/2.develop/lake/structures/transaction.mdx b/docs/2.develop/lake/structures/transaction.mdx new file mode 100644 index 00000000000..5c0b8b72261 --- /dev/null +++ b/docs/2.develop/lake/structures/transaction.mdx @@ -0,0 +1,161 @@ +--- +sidebar_position: 5 +sidebar_label: Transaction +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# `Transaction` Structure + +## Definition + +Transaction is the main way of interraction between a user and a blockchain. Transaction contains: +- Signer account ID +- Receiver account ID +- Actions + +## `SignedTransactionView` + +Transaction might be unsigned but from the indexer perspective when we think about Transaction we think about signed one + + + + +```rust links=1 +pub struct SignedTransactionView { + pub signer_id: AccountId, + pub public_key: PublicKey, + pub nonce: Nonce, + pub receiver_id: AccountId, + pub actions: Vec, + pub signature: Signature, + pub hash: CryptoHash, +} +``` + + + + +```ts links=1 +export type Transaction = { + signerId: string; + publicKey: string; + nonce: number; + receiverId: string; + actions: Action[]; + signature: string; + hash: string; +}; +``` + + + + + +## `ActionView` + +`ActionView` is an Enum with possible actions along with parameters. This structure is used in Transactions and in [Receipts](./receipt.mdx) + + + + +```rust links=1 +pub enum ActionView { + CreateAccount, + DeployContract { + code: String, + }, + FunctionCall { + method_name: String, + args: String, + gas: Gas, + #[serde(with = "u128_dec_format")] + deposit: Balance, + }, + Transfer { + #[serde(with = "u128_dec_format")] + deposit: Balance, + }, + Stake { + #[serde(with = "u128_dec_format")] + stake: Balance, + public_key: PublicKey, + }, + AddKey { + public_key: PublicKey, + access_key: AccessKeyView, + }, + DeleteKey { + public_key: PublicKey, + }, + DeleteAccount { + beneficiary_id: AccountId, + }, +} +``` + + + + +```ts links=1 +export type Action = + | "CreateAccount" + | DeployContractAction + | FunctionCallAction + | TransferAction + | StakeAction + | AddKeyAction + | DeleteKeyAction + | DeleteAccountAction; + +export type DeployContractAction = { + DeployContract: { + code: string; + }; +}; + +export type FunctionCallAction= { + FunctionCall: { + methodName: string; + args: string; + gas: number; + deposit: string; + }; +}; + +export type TransferAction = { + Transfer: { + deposit: string; + }; +}; + +export type StakeAction = { + Stake: { + stake: number; + publicKey: string; + }; +}; + +export type AddKeyAction = { + AddKey: { + publicKey: string; + accessKey: AccessKey; + }; +}; + +export type DeleteKeyAction = { + DeleteKey: { + publicKey: string; + }; +}; + +export type DeleteAccountAction = { + DeleteAccount: { + beneficiaryId: string; + }; +}; +``` + + + diff --git a/website/sidebars.json b/website/sidebars.json index f119f992ca3..6be0fda1f2d 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -54,7 +54,11 @@ }, [ { - "Indexers": ["concepts/advanced/indexers", "concepts/advanced/near-indexer-framework", "concepts/advanced/near-lake-framework"] + "Indexers": [ + "concepts/advanced/indexers", + "concepts/advanced/near-indexer-framework", + "concepts/advanced/near-lake-framework" + ] }, "concepts/basics/runtime", "concepts/advanced/specification", @@ -216,6 +220,18 @@ } ] }, + { + "Lake Indexer": [ + "develop/lake/structures/toc", + "develop/lake/structures/block", + "develop/lake/structures/chunk", + "develop/lake/structures/shard", + "develop/lake/structures/transaction", + "develop/lake/structures/receipt", + "develop/lake/structures/execution-outcome", + "develop/lake/structures/state-change" + ] + }, { "type": "link", "label": "RPC API ↗", From 07211828f69c066c96b2d43e5d5d0602ce478f51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dami=C3=A1n=20Parrino?= Date: Thu, 10 Aug 2023 15:02:26 -0300 Subject: [PATCH 5/5] Update indexer4explorer.md --- docs/4.tools/indexer4explorer.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/docs/4.tools/indexer4explorer.md b/docs/4.tools/indexer4explorer.md index 1bacf0838e0..b3ab4ebbd97 100644 --- a/docs/4.tools/indexer4explorer.md +++ b/docs/4.tools/indexer4explorer.md @@ -32,32 +32,6 @@ For a reliable source of real-time and past information, please run your own [in --- -## Connecting to the Database - -You can use any database manager compatible with PostgreSQL. If you don't know any, we can recommend you to try [DBeaver Community](https://dbeaver.io/). - -| Network | Host | Port | Database | Username | Password | -| ------- | ------------------------------------ | ---- | ---------------- | --------------- | ------------ | -| mainnet | mainnet.db.explorer.indexer.near.dev | 5432 | mainnet_explorer | public_readonly | nearprotocol | -| testnet | testnet.db.explorer.indexer.near.dev | 5432 | testnet_explorer | public_readonly | nearprotocol | - - - - - - postgres://public_readonly:nearprotocol@testnet.db.explorer.indexer.near.dev/testnet_explorer - - - - - - postgres://public_readonly:nearprotocol@mainnet.db.explorer.indexer.near.dev/mainnet_explorer - - - - ---- - ## Example Queries ### Transactions Calling a Method Query for all transactions that called `contribute` in the `v1.faucet.nonofficial.testnet` testnet account.