Skip to content

Commit

Permalink
Update data-storage.md
Browse files Browse the repository at this point in the history
  • Loading branch information
bucanero authored Jul 24, 2023
1 parent ebbae31 commit e4d3b6e
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion docs/1.concepts/data-flow/data-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ As you can imagine, this basic block information might not be enough for a compl
More advanced real-world queries and operations like aggregation, search, relationships, and non-trivial filtering are simply not possible using RPC calls alone. An example of this is showing a leaderboard of the most popular NFT collections in the last 7 days. It would not be possible to get that information by interacting directly with the contract itself or via RPC. Getting this data would require processing every single transaction that ever happened, aggregating them in terms of collection, filtering them by timestamps calculation, and ordering them in terms of total transacted value.


One option to solve these complex data requirements is to build a server using an indexing framework. A blockchain indexer is an implementation of the push model of getting the data. Instead of actively pulling the data from the source, indexers listen to the stream of data from the blockchain, and the data can be immediately filtered and processed according to defined requirements. Indexers can also be used to simplify the "wide" query execution. For example, a stream of data can be written to a permanent database for later data analysis using a convenient query language like SQL.
One option to solve these complex data requirements is to build a server using an indexing framework. A blockchain indexer is an implementation of the push model of getting the data. Instead of actively pulling the data from the source, indexers listen to the stream of data from the blockchain, and the data can be immediately filtered and processed according to defined requirements. Indexers can also be used to simplify the "wide" query execution. For example, a data stream can be written to a permanent database for later data analysis using a convenient query language like SQL.


## Data tools
Expand All @@ -83,6 +83,7 @@ As an alternative, you can try one of the many available indexing services on th
* [NEAR Indexer for Explorer](/tools/indexer-for-explorer): leverages the indexer micro-framework to watch and store all of the blockchain's events/data into a transactional PostgreSQL database. You can clone the [GitHub repository](https://github.com/near/near-indexer-for-explorer) and customize your own indexer solution.
* [NEAR Lake Framework](/concepts/advanced/near-lake-framework): a companion library to NEAR Lake. It allows you to build your own indexer that watches a stream of blocks from a NEAR Lake data source and allows you to create your own logic to process that data. Keep in mind this is probably the one you want to use for future projects, instead of the Indexer Framework. Read [why is better](/concepts/advanced/near-indexer-framework#why-is-it-better-than-near-indexer-framework).
* [NEAR Lake Indexer](/concepts/advanced/near-lake-framework): leverages the indexer micro-framework to watch and store all of the blockchain's events/data as JSON files on a user-specified AWS S3 or S3-compatible storage.
* [Near Query API](https://near.org/dataplatform.near/widget/QueryApi.Dashboard): the QueryAPI allows you to seamlessly create, manage, and discover indexers on the Blockchain Operating System (BOS). Developers can deploy their indexers to aggregate historical data in a matter of minutes, while accessing the data via custom GraphQL queries.
* [The Graph](https://thegraph.com/docs/en/cookbook/near/): The Graph gives developers tools to process blockchain events and make the resulting data easily available via a GraphQL API, known individually as a subgraph. [Graph Node](https://github.com/graphprotocol/graph-node) is now able to process NEAR events, which means that NEAR developers can now build subgraphs to index their smart contracts.
* [Pagoda API](https://pagoda.co) is a hosted service managed by Pagoda that solves the complex data problem by indexing the blockchain into datamarts and is accessible via standard RESTful APIs for you to directly use in any part of your application. The API enables performant and efficient querying of blockchain data without having to build and manage your own indexer infrastructure. Pagoda also offers the Indexer framework as a decentralized solution to indexing on-chain data, and a centralized Data Lake source that can be used to build your own indexers and servers for such data query needs.
* [PIKESPEAK API](https://pikespeak.ai): an enterprise-grade API where you can fetch blockchain events and aggregated analytics on wallets, validators, delegators, money transfers, dapps activity and more. [Documentation](https://doc.pikespeak.ai/)
Expand Down

0 comments on commit e4d3b6e

Please sign in to comment.