-
Notifications
You must be signed in to change notification settings - Fork 360
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* query api docs
- Loading branch information
Showing
7 changed files
with
259 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,170 @@ | ||
--- | ||
id: index-functions | ||
title: Indexing Functions | ||
sidebar_label: Indexing Functions | ||
--- | ||
|
||
:::info | ||
QueryAPI is a fully managed service that allows you to create and manage indexers on-chain seamlessly. | ||
::: | ||
|
||
|
||
## Indexing | ||
|
||
Let's review a [very simple indexer](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=roshaan.near/demo-indexer&view=editor-window), which will help you to understand | ||
how the indexer's indexing logic works. | ||
|
||
```js title=indexingLogic.js | ||
import { Block } from "@near-lake/primitives"; | ||
|
||
/** | ||
* getBlock(block, context) applies your custom logic to a Block on Near and commits the data to a database. | ||
* | ||
* @param {block} Block - A Near Protocol Block | ||
* @param {context} - A set of helper methods to retrieve and commit state | ||
*/ | ||
async function getBlock(block: Block, context) { | ||
const h = block.header().height; | ||
await context.set("height", h); | ||
} | ||
``` | ||
|
||
In the `getBlock()` function, you're given a block, which is a block on the Near blockchain, as | ||
well as a `context` object, which gives you a set of helper methods to be able to commit | ||
what you want to the database that QueryAPI has provisioned for you. | ||
|
||
The code is going into the header of the `block` | ||
and getting the block's `height`, and then is using the `context` object to set a key value store. | ||
|
||
Next, if you check out the database schema: | ||
|
||
```sql title=schema.sql | ||
CREATE TABLE | ||
"indexer_storage" ( | ||
"function_name" TEXT NOT NULL, | ||
"key_name" TEXT NOT NULL, | ||
"value" TEXT NOT NULL, | ||
PRIMARY KEY ("function_name", "key_name") | ||
) | ||
``` | ||
|
||
It's a very simple schema where you can store the `function_name`, `key_name`, and `value`. | ||
|
||
:::tip | ||
That's all this indexer function is doing: it sets the `height` value equal to the current block's height. | ||
::: | ||
|
||
|
||
<!-- ![QueryAPI Indexer Dashboard](/docs/assets/QAPIScreen2.png) --> | ||
|
||
|
||
## Local Debug Mode | ||
|
||
While you're iterating on your indexer development, it's critical to have some type of debugging | ||
functionality to be able to test with, and the _Debug Mode_ is very helpful for that. | ||
|
||
![QueryAPI Dashboard](/docs/assets/QAPIdebug.png) | ||
|
||
For example, if you want to test the [simple indexer](#indexing) explained in the previous section | ||
using the local debugging mode: | ||
|
||
- Enable <kbd>Debug Mode</kbd> on the **Indexer Editor** | ||
- Add a block to your debug list (e.g., `97779559`) | ||
- Go into your web browser's Console | ||
- Finally, click <kbd>Play</kbd>. | ||
|
||
|
||
On your browser's Console, you should see the indexer's debug execution where it sets the `height` key to `97779559`: | ||
|
||
![QueryAPI Indexer Dashboard](/docs/assets/QAPIdebuglog.png) | ||
|
||
:::info Local tests | ||
All debug mode tests are happening **locally**, so they do not reach the database. | ||
All your queries and mutations will return empty objects. | ||
::: | ||
|
||
:::tip | ||
You can also click <kbd>Follow the Network</kbd> and it will show how your indexer logic works throughout. | ||
::: | ||
|
||
## Contract filters | ||
|
||
A contract filter is used by QueryAPI to do backend optimizations to | ||
help do historical indexing faster. | ||
While creating an indexer, when you define a contract filter, | ||
QueryAPI will send any single transaction event that passes this filter to your indexer function | ||
so you can index it. | ||
|
||
If you only want to index events from a single contract, simply define the contract name on the **Contract Filter** text box. | ||
In some cases you might want to either support indexing on multiple contracts, | ||
or simply support every single contract that exists on the Near blockchain. | ||
|
||
#### Single contract filter | ||
|
||
|
||
For example, if you check out the [simple indexer](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=roshaan.near/demo-indexer&view=editor-window), you'll see that in this case | ||
you have a `social.near` contract filter. | ||
In this example, the indexer is only concerned on indexing events from `social.near`'s contract. | ||
|
||
#### Multiple contracts filter | ||
|
||
For example, if you want to index all the contracts from AstroDAO, where there is an account created | ||
for each and every different DAO, you should define `*.sputnik-dao.near` as the contract filter. | ||
Likewise, if you want to get events from every contract on the blockchain, simply define `*` as the filter. | ||
|
||
## Feed-indexer logic | ||
|
||
Then we call context.graphql, which allows us to make arbitrary mutations and queries | ||
to our database that we provision for you. | ||
If you're interested in how to create GraphQL queries, there's a whole bunch of resources | ||
online. | ||
In this case, we are passing in our mutation data, which has a post object, and it's inserting | ||
it inside Postgres, I mean, inside of Postgres using GraphQL. | ||
But it's very easy to create these mutations. | ||
|
||
## Mutations in GraphQL | ||
|
||
If you go to the GraphiQL tab, you can access the GraphiQL Explorer that provides a user friendly GraphQL playground, where you can view and create queries and mutations based on the DB schema that you defined for the indexer. | ||
|
||
![QueryAPI Indexer Dashboard](/docs/assets/QAPIgraphiql.png) | ||
|
||
You can easily set some fields and select the returning data | ||
that you want, and the tool will build a query on the mutation panel on the right. | ||
Then you can copy the resulting query, either in your JavaScript code so that you pass actual | ||
data manually, or you pass in the mutation data object as a second parameter. | ||
|
||
For example, if you go and add a new mutation, click <kbd>+</kbd>, then you can do a bunch of actions here, such as creating, deleting, or inserting posts into your table. | ||
|
||
![Playground](/docs/assets/QAPIScreen.gif) | ||
|
||
If you want to test your mutation, using [Debug Mode](#local-debug-mode) you can add a specific | ||
block to the list, and then play it to see how it works. | ||
Based on the indexer logic you defined, you'll get a call to the GraphQL mutation with the object | ||
and data passed into it. | ||
|
||
:::tip Video Walkthrough | ||
|
||
**Tip:** watch the video on how to [create mutations in GraphQL](https://www.youtube.com/watch?v=VwO6spk8D58&t=781s). | ||
|
||
::: | ||
|
||
|
||
## Create a BOS component from query | ||
|
||
Creating a BOS component from a GraphQL query is simple when using the GraphQL Playground. Just follow these steps: | ||
|
||
- go to the GraphiQL tab | ||
- select the query that you want to use | ||
- click on the <kbd>Show GraphiQL Code Exporter</kbd> button | ||
- get some default code here, copy it, | ||
- go to the BOS sandbox, paste it. | ||
|
||
|
||
This will set up some boilerplate code to execute the GraphQL query, add the query that you had | ||
in your playground and then call that query, extract the data and render it using the | ||
render data function. | ||
|
||
Once you have the BOS component code, you can test it out by going to [the sandbox](https://near.org/sandbox), | ||
pasting the generated code, and then selecting <kbd>Component Preview</kbd>. | ||
Next, you can create a nice UI over this boilerplate code, and publish your new BOS component. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
--- | ||
id: intro | ||
title: QueryAPI Overview | ||
sidebar_label: Introduction | ||
--- | ||
|
||
|
||
Near QueryAPI is a fully managed solution to build indexer functions, | ||
extract on-chain data, store it in a database, and be able to query it using GraphQL endpoints. | ||
|
||
## Indexing complexity | ||
|
||
Blockchain Indexers are known to be difficult to create, maintain, and operate. | ||
You have to focus on the business logic of your indexer, while you also have to | ||
take care of everything else around it. | ||
A dedicated team member could be needed to deal with all these challenges. | ||
|
||
Common indexing challenges include: | ||
|
||
#### Creation | ||
|
||
- Design Database Schema and provision it with correct configurations for security, data retention, and performance. | ||
- Write and test indexer code that interacts with the database | ||
- Deploy Indexer to a Cloud provider. Ensure network permissions firewalls, PCs, or other network-related settings are setup correctly. | ||
- Create an API endpoint to retrieve data from your database for your fronted applications | ||
|
||
#### Maintenance | ||
|
||
- Monitor performance of your database and scale it as needed | ||
- Manage permissions and access to database with changing requirements | ||
|
||
#### Operation | ||
|
||
- Re-index data due to issues and updates. Ensuring that production environments don't get disrupted. | ||
- Perform database schema migrations | ||
- Scale the API as your application grows | ||
- Keep up with all the underlying blockchain nodes and upgrades | ||
|
||
|
||
## QueryAPI | ||
|
||
As you can see, running indexers is a complex and comprehensive set of processes, and | ||
Near QueryAPI tries to cover most (or all) of these needs offering an open-source solution for creating, managing, and exploring indexers. | ||
|
||
### BOS Component | ||
|
||
QueryAPI has a [`QueryApi.App` BOS widget](https://near.org/#/dataplatform.near/widget/QueryApi.App), hosted under the `dataplatform.near` account. | ||
With this component you can see all the public indexers currently available on the Near blockchain. | ||
|
||
If you would like to create a new indexer, simply click [**Create New Indexer**](https://near.org/#/dataplatform.near/widget/QueryApi.App/?view=create-new-indexer). | ||
|
||
![QueryAPI Dashboard](/docs/assets/QAPIScreen.png) | ||
|
||
### Indexers stored on-chain | ||
|
||
QueryAPI stores all the indexer logic and schemas used to provision the databases on-chain. | ||
Whenever you interact with the QueryAPI BOS component, in the background it's making an RPC query to [`queryapi.dataplatform.near`](https://stats.gallery/mainnet/queryapi.dataplatform.near/contract?t=week), | ||
where a smart contract stores all of your indexer logic as well as your schemas. | ||
|
||
For example, if you select the _feed-indexer_ and click on [**View indexer**](https://near.org/dataplatform.near/widget/QueryApi.App?selectedIndexerPath=dataplatform.near/feed-indexer&view=editor-window) you'll see all the details about an indexer that powers the [near.org](https://near.org)'s main posts feed. | ||
You're free to review the JavaScript code of the indexer function, or check the SQL that defines the database schema. | ||
|
||
|
||
## Known limitations | ||
|
||
- Currently under closed beta-testing. | ||
- Only supports JavaScript indexers. (we plan to support Rust in the future) | ||
- It always takes the latest `@near-lake/primitives` library. | ||
- It doesn't support schema migrations. | ||
- If you have an indexer whose schema needs to change you may need to create a new indexer and do historical backfill on that new indexer again. | ||
- No way to stop your indexer or restart it truncating all tables. | ||
- Historical backfill works in parallel to the real-time indexing. | ||
- Keep that in mind just to be sure that you don't have unintended side effects. | ||
- Pagoda currently doesn't charge for storage of your indexer code and data as well as running the indexer, but we will introduce this soon. | ||
|
||
:::tip Join the Beta | ||
|
||
If you would like to be part of the closed beta please [fill out this form](https://near.org/dev-queryapi.dataplatform.near/widget/NearQueryApi) or [reach out on Telegram](https://nearbuilders.com/tg-data) for access. | ||
|
||
::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.