TPScore simplifies TPS Data Analysis for non-technical users in the Polkadot Ecosystem.
TPScore aims to provide non-technical users in the Polkadot ecosystem with an accessible and user-friendly platform for analyzing TPS (Transactions per Second) data. Our goal is to bridge the gap between technical intricacies and user-friendly visualization, empowering individuals to make informed decisions about blockchain adoption, investment, and development.
The project consists of two main parts:
- ETL System
- User Interface (UI) with Data Visualizations
This repository contains the ETL system, responsible for collecting blockchain data from Polkadot parachains, calculating the Transactions Per Second (TPS) for transactions of type Balance, and storing the processed data in a MySQL database. This repository is a part of Milestone 1 submission for the Grants-Program.
Before running this project, make sure you have the following prerequisites installed:
Follow these steps to set up and run the project:
Clone the Project
- Clone this repository using
git clonecommand.
Create and Run MySQL Database Container
- Navigate to the 'db' directory from the project's root directory:
cd db. - Build the Docker image using the Dockerfile:
docker build -t mysql_tpscore . - Run a container with the MySQL database:
docker run --name db_mysql -d mysql_tpscore
Create and Run Airflow Container
- Navigate to the 'airflow' directory from the project's root directory:
cd airflow. - Launch the airflow-init container:
docker compose up airflow-init - Build an image using the Dockerfile and docker-compose files:
docker build . - Start Airflow services:
docker compose up -d
Create a Network and Add Containers
- Create a Docker network:
docker network create net_tpscore - Connect the 'db_mysql' container:
docker network connect net_tpscore db_mysql - Connect the 'airflow_scheduler' container:
docker network connect net_tpscore airflow_scheduler
The application can be interacted with in two ways:
Airflow GUI
- Access the Airflow GUI at http://localhost:8080/
- Log in using the credentials (airflow, airflow)
- Find the
get_data_tpscoreDAG and activate it by moving toggle to the right
- Click on the DAG to verify successful execution (squares will be dark green, indicating error-free execution).
MySQL Connection from Inside the Container
- Connect to the container:
docker exec -it db_mysql bash - Log in to MySQL:
mysql -u user_tpscore -p - Enter the password:
pass_tpscore - Select the database:
USE tpscore_data; - Run queries on the MySQL database, for example, select the top 3 TPS values:
SELECT * FROM tps ORDER BY tps DESC LIMIT 3;
The core logic of the application is in get_data_tpscore.py. The script defines functions for database connection, fetching blockchain data, transformation, and data upload to the database. Tests for get_data_tpscore.py are in test_get_data_tpscore.py. Follow these steps to run the tests:
- Create a virtual environment by running the command from the root project directory:
python -m venv venv_tpscore - Activate the environment:
source venv_tpscore/bin/activate - Install dependencies from
requirements.txt:pip install -r requirements.txt - Run tests:
pytest
This open-source project was developed for the Polkadot community and relies on various open-source components and Python libraries:
This project is licensed under GPLv3. For more details, refer to the license document.