Processes raw indexer-service logs from a RabbitMQ queue, saves the results to a PostgreSQL database.
The processing:
- Extracts details about query execution form the log lines: The subgraph deployment, timestamp, query, variables, execution time and fees paid.
- Normalizes the queries, separates the "query skeleton" from all values and variables.
- Saves the query skeleton (if never seen before), as well as a logs all the information about the query execution (including the variable values used).
Refer to AutoAgora for more details.
Just build the container!
docker build -t autoagora-processor .Because it is a stateless service, and only consumes from a RabbitMQ queue and adds rows to a PostgreSQL database, the Autoagora Processor can safely be dynamically scaled horizontally. Note however that it has been observed processing upwards of 50 queries per second on a single CPU core.
Note that it also needs to connect to the indexer-agent's management API to determine the indexer's current
allocations, and then pull their schemas using the graph-node query endpoint.
Configuration:
usage: autoagora-processor [-h] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] --rabbitmq-host
RABBITMQ_HOST [--rabbitmq-queue-name RABBITMQ_QUEUE_NAME]
[--rabbitmq-queue-limit RABBITMQ_QUEUE_LIMIT]
[--rabbitmq-exchange-name RABBITMQ_EXCHANGE_NAME]
[--rabbitmq-username RABBITMQ_USERNAME]
[--rabbitmq-password RABBITMQ_PASSWORD] --postgres-host POSTGRES_HOST
--postgres-database POSTGRES_DATABASE --postgres-username
POSTGRES_USERNAME --postgres-password POSTGRES_PASSWORD
[--postgres-port POSTGRES_PORT] --graph-node-query-endpoint
GRAPH_NODE_QUERY_ENDPOINT --indexer-agent-mgmt-endpoint
INDEXER_AGENT_MGMT_ENDPOINT
options:
-h, --help show this help message and exit
--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
[env var: LOG_LEVEL] (default: WARNING)
--rabbitmq-host RABBITMQ_HOST
Hostname of the RabbitMQ server used for queuing the GQL logs. [env var:
RABBITMQ_HOST] (default: None)
--rabbitmq-queue-name RABBITMQ_QUEUE_NAME
Name of the RabbitMQ queue to pull query-node logs from. [env var:
RABBITMQ_QUEUE_NAME] (default: gql_logs_processor)
--rabbitmq-queue-limit RABBITMQ_QUEUE_LIMIT
Size limit of the created RabbitMQ queue. It is discouraged to change that
value while the system is running, because it requires manual destruction
of the queue and a restart of the whole Auto Agora stack. [env var:
RABBITMQ_QUEUE_LIMIT] (default: 1000)
--rabbitmq-exchange-name RABBITMQ_EXCHANGE_NAME
Name of the RabbitMQ exchange query-node logs are pushed to. [env var:
RABBITMQ_EXCHANGE_NAME] (default: gql_logs)
--rabbitmq-username RABBITMQ_USERNAME
Username to use for the GQL logs RabbitMQ queue. [env var:
RABBITMQ_USERNAME] (default: guest)
--rabbitmq-password RABBITMQ_PASSWORD
Password to use for the GQL logs RabbitMQ queue. [env var:
RABBITMQ_PASSWORD] (default: guest)
--postgres-host POSTGRES_HOST
Host of the postgres instance storing the logs. [env var: POSTGRES_HOST]
(default: None)
--postgres-database POSTGRES_DATABASE
Name of the logs database. [env var: POSTGRES_DATABASE] (default: None)
--postgres-username POSTGRES_USERNAME
Username for the logs database. [env var: POSTGRES_USERNAME] (default:
None)
--postgres-password POSTGRES_PASSWORD
Password for the logs database. [env var: POSTGRES_PASSWORD] (default:
None)
--postgres-port POSTGRES_PORT
Port for the logs database. [env var: POSTGRES_PORT] (default: 5432)
--graph-node-query-endpoint GRAPH_NODE_QUERY_ENDPOINT
URL of the indexer's graph-node GraphQL query endpoint. [env var:
GRAPH_NODE_QUERY_ENDPOINT] (default: None)
--indexer-agent-mgmt-endpoint INDEXER_AGENT_MGMT_ENDPOINT
URL to the indexer-agent management GraphQL endpoint. [env var:
INDEXER_AGENT_MGMT_ENDPOINT] (default: None)