diff --git a/docs/bos/community/indexers.md b/docs/bos/community/indexers.md index 79533fa9ac5..7585a5349a6 100644 --- a/docs/bos/community/indexers.md +++ b/docs/bos/community/indexers.md @@ -1,7 +1,7 @@ --- id: indexers -title: QueryAPI Indexers -sidebar_label: QueryAPI +title: QueryAPI Indexing Example +sidebar_label: QueryAPI Example --- With QueryAPI you can quickly create your own indexer by: @@ -24,7 +24,7 @@ You can request access through [this link](https://near.org/dev-queryapi.datapla ::: -### How it Works +## How it Works This works by: @@ -147,7 +147,7 @@ The GraphiQL tab in the editor will allow you to view the returned data from you ::: -### Performing Queries on the Public GraphQL API +## Performing Queries on the Public GraphQL API In this section, we will provide a brief overview of how to query from a component in BOS. diff --git a/docs/bos/queryapi/index-function.md b/docs/bos/queryapi/index-function.md new file mode 100644 index 00000000000..b5f3c30f9aa --- /dev/null +++ b/docs/bos/queryapi/index-function.md @@ -0,0 +1,170 @@ +--- +id: index-functions +title: Indexing Functions +sidebar_label: Indexing Functions +--- + +:::info +QueryAPI is a fully managed service that allows you to create and manage indexers on-chain seamlessly. +::: + + +## Indexing + +Let's review a [very simple indexer](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=roshaan.near/demo-indexer&view=editor-window), which will help you to understand +how the indexer's indexing logic works. + +```js title=indexingLogic.js +import { Block } from "@near-lake/primitives"; + +/** + * getBlock(block, context) applies your custom logic to a Block on Near and commits the data to a database. + * + * @param {block} Block - A Near Protocol Block + * @param {context} - A set of helper methods to retrieve and commit state + */ +async function getBlock(block: Block, context) { + const h = block.header().height; + await context.set("height", h); +} +``` + +In the `getBlock()` function, you're given a block, which is a block on the Near blockchain, as +well as a `context` object, which gives you a set of helper methods to be able to commit +what you want to the database that QueryAPI has provisioned for you. + +The code is going into the header of the `block` +and getting the block's `height`, and then is using the `context` object to set a key value store. + +Next, if you check out the database schema: + +```sql title=schema.sql +CREATE TABLE + "indexer_storage" ( + "function_name" TEXT NOT NULL, + "key_name" TEXT NOT NULL, + "value" TEXT NOT NULL, + PRIMARY KEY ("function_name", "key_name") + ) +``` + +It's a very simple schema where you can store the `function_name`, `key_name`, and `value`. + +:::tip +That's all this indexer function is doing: it sets the `height` value equal to the current block's height. +::: + + + + + +## Local Debug Mode + +While you're iterating on your indexer development, it's critical to have some type of debugging +functionality to be able to test with, and the _Debug Mode_ is very helpful for that. + +![QueryAPI Dashboard](/docs/assets/QAPIdebug.png) + +For example, if you want to test the [simple indexer](#indexing) explained in the previous section +using the local debugging mode: + +- Enable Debug Mode on the **Indexer Editor** +- Add a block to your debug list (e.g., `97779559`) +- Go into your web browser's Console +- Finally, click Play. + + +On your browser's Console, you should see the indexer's debug execution where it sets the `height` key to `97779559`: + +![QueryAPI Indexer Dashboard](/docs/assets/QAPIdebuglog.png) + +:::info Local tests +All debug mode tests are happening **locally**, so they do not reach the database. +All your queries and mutations will return empty objects. +::: + +:::tip +You can also click Follow the Network and it will show how your indexer logic works throughout. +::: + +## Contract filters + +A contract filter is used by QueryAPI to do backend optimizations to +help do historical indexing faster. +While creating an indexer, when you define a contract filter, +QueryAPI will send any single transaction event that passes this filter to your indexer function +so you can index it. + +If you only want to index events from a single contract, simply define the contract name on the **Contract Filter** text box. +In some cases you might want to either support indexing on multiple contracts, +or simply support every single contract that exists on the Near blockchain. + +#### Single contract filter + + +For example, if you check out the [simple indexer](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=roshaan.near/demo-indexer&view=editor-window), you'll see that in this case +you have a `social.near` contract filter. +In this example, the indexer is only concerned on indexing events from `social.near`'s contract. + +#### Multiple contracts filter + +For example, if you want to index all the contracts from AstroDAO, where there is an account created +for each and every different DAO, you should define `*.sputnik-dao.near` as the contract filter. +Likewise, if you want to get events from every contract on the blockchain, simply define `*` as the filter. + +## Feed-indexer logic + +Then we call context.graphql, which allows us to make arbitrary mutations and queries +to our database that we provision for you. +If you're interested in how to create GraphQL queries, there's a whole bunch of resources +online. +In this case, we are passing in our mutation data, which has a post object, and it's inserting +it inside Postgres, I mean, inside of Postgres using GraphQL. +But it's very easy to create these mutations. + +## Mutations in GraphQL + +If you go to the GraphiQL tab, you can access the GraphiQL Explorer that provides a user friendly GraphQL playground, where you can view and create queries and mutations based on the DB schema that you defined for the indexer. + +![QueryAPI Indexer Dashboard](/docs/assets/QAPIgraphiql.png) + +You can easily set some fields and select the returning data +that you want, and the tool will build a query on the mutation panel on the right. +Then you can copy the resulting query, either in your JavaScript code so that you pass actual +data manually, or you pass in the mutation data object as a second parameter. + +For example, if you go and add a new mutation, click +, then you can do a bunch of actions here, such as creating, deleting, or inserting posts into your table. + +![Playground](/docs/assets/QAPIScreen.gif) + +If you want to test your mutation, using [Debug Mode](#local-debug-mode) you can add a specific +block to the list, and then play it to see how it works. +Based on the indexer logic you defined, you'll get a call to the GraphQL mutation with the object +and data passed into it. + +:::tip Video Walkthrough + +**Tip:** watch the video on how to [create mutations in GraphQL](https://www.youtube.com/watch?v=VwO6spk8D58&t=781s). + +::: + + +## Create a BOS component from query + +Creating a BOS component from a GraphQL query is simple when using the GraphQL Playground. Just follow these steps: + +- go to the GraphiQL tab +- select the query that you want to use +- click on the Show GraphiQL Code Exporter button +- get some default code here, copy it, +- go to the BOS sandbox, paste it. + + +This will set up some boilerplate code to execute the GraphQL query, add the query that you had +in your playground and then call that query, extract the data and render it using the +render data function. + +Once you have the BOS component code, you can test it out by going to [the sandbox](https://near.org/sandbox), +pasting the generated code, and then selecting Component Preview. +Next, you can create a nice UI over this boilerplate code, and publish your new BOS component. + diff --git a/docs/bos/queryapi/intro.md b/docs/bos/queryapi/intro.md new file mode 100644 index 00000000000..79989a22e6e --- /dev/null +++ b/docs/bos/queryapi/intro.md @@ -0,0 +1,80 @@ +--- +id: intro +title: QueryAPI Overview +sidebar_label: Introduction +--- + + +Near QueryAPI is a fully managed solution to build indexer functions, +extract on-chain data, store it in a database, and be able to query it using GraphQL endpoints. + +## Indexing complexity + +Blockchain Indexers are known to be difficult to create, maintain, and operate. +You have to focus on the business logic of your indexer, while you also have to +take care of everything else around it. +A dedicated team member could be needed to deal with all these challenges. + +Common indexing challenges include: + +#### Creation + +- Design Database Schema and provision it with correct configurations for security, data retention, and performance. +- Write and test indexer code that interacts with the database +- Deploy Indexer to a Cloud provider. Ensure network permissions firewalls, PCs, or other network-related settings are setup correctly. +- Create an API endpoint to retrieve data from your database for your fronted applications + +#### Maintenance + +- Monitor performance of your database and scale it as needed +- Manage permissions and access to database with changing requirements + +#### Operation + +- Re-index data due to issues and updates. Ensuring that production environments don't get disrupted. +- Perform database schema migrations +- Scale the API as your application grows +- Keep up with all the underlying blockchain nodes and upgrades + + +## QueryAPI + +As you can see, running indexers is a complex and comprehensive set of processes, and +Near QueryAPI tries to cover most (or all) of these needs offering an open-source solution for creating, managing, and exploring indexers. + +### BOS Component + +QueryAPI has a [`QueryApi.App` BOS widget](https://near.org/#/dataplatform.near/widget/QueryApi.App), hosted under the `dataplatform.near` account. +With this component you can see all the public indexers currently available on the Near blockchain. + +If you would like to create a new indexer, simply click [**Create New Indexer**](https://near.org/#/dataplatform.near/widget/QueryApi.App/?view=create-new-indexer). + +![QueryAPI Dashboard](/docs/assets/QAPIScreen.png) + +### Indexers stored on-chain + +QueryAPI stores all the indexer logic and schemas used to provision the databases on-chain. +Whenever you interact with the QueryAPI BOS component, in the background it's making an RPC query to [`queryapi.dataplatform.near`](https://stats.gallery/mainnet/queryapi.dataplatform.near/contract?t=week), +where a smart contract stores all of your indexer logic as well as your schemas. + +For example, if you select the _feed-indexer_ and click on [**View indexer**](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=dataplatform.near/feed-indexer&view=editor-window) you'll see all the details about an indexer that powers the [near.org](https://near.org)'s main posts feed. +You're free to review the JavaScript code of the indexer function, or check the SQL that defines the database schema. + + +## Known limitations + +- Currently under closed beta-testing. +- Only supports JavaScript indexers. (we plan to support Rust in the future) +- It always takes the latest `@near-lake/primitives` library. +- It doesn't support schema migrations. + - If you have an indexer whose schema needs to change you may need to create a new indexer and do historical backfill on that new indexer again. +- No way to stop your indexer or restart it truncating all tables. +- Historical backfill works in parallel to the real-time indexing. + - Keep that in mind just to be sure that you don't have unintended side effects. +- Pagoda currently doesn't charge for storage of your indexer code and data as well as running the indexer, but we will introduce this soon. + +:::tip Join the Beta + +If you would like to be part of the closed beta please [fill out this form](https://near.org/dev-queryapi.dataplatform.near/widget/NearQueryApi) or [reach out on Telegram](https://nearbuilders.com/tg-data) for access. + +::: \ No newline at end of file diff --git a/website/sidebars.json b/website/sidebars.json index 7832d2de4b2..6d5b02dedaa 100644 --- a/website/sidebars.json +++ b/website/sidebars.json @@ -581,7 +581,11 @@ ] }, { - "Indexing Tools": ["bos/community/indexers"] + "QueryAPI Indexing": [ + "bos/queryapi/intro", + "bos/queryapi/index-functions", + "bos/community/indexers" + ] }, { "type": "html", diff --git a/website/static/docs/assets/QAPIdebug.png b/website/static/docs/assets/QAPIdebug.png new file mode 100644 index 00000000000..6c6d70c574b Binary files /dev/null and b/website/static/docs/assets/QAPIdebug.png differ diff --git a/website/static/docs/assets/QAPIdebuglog.png b/website/static/docs/assets/QAPIdebuglog.png new file mode 100644 index 00000000000..a78fc047e07 Binary files /dev/null and b/website/static/docs/assets/QAPIdebuglog.png differ diff --git a/website/static/docs/assets/QAPIgraphiql.png b/website/static/docs/assets/QAPIgraphiql.png new file mode 100644 index 00000000000..d4dda4fb5fc Binary files /dev/null and b/website/static/docs/assets/QAPIgraphiql.png differ