fix(blobstorage): fix command instructions and missing local_docker f…

…older in packages/blobstorage (#16464)

Co-authored-by: jeff <[email protected]>
Co-authored-by: Roger <[email protected]>

packages/blobstorage/.default.indexer.env

@@ -2,8 +2,8 @@ RPC_URL=wss://l1ws.internal.taiko.xyz
 BEACON_URL=https://l1beacon.internal.taiko.xyz/eth/v1/beacon/blob_sidecars/
 TAIKO_L1_CONTRACT_ADDRESS=0xC069c3d2a9f2479F559AD34485698ad5199C555f
 DATABASE_HOST=localhost
-DATABASE_PORT=27017
+DATABASE_PORT=3306
 DATABASE_USER=root
-DATABASE_PASSWORD=root
+DATABASE_PASSWORD=passw00d
 DATABASE_NAME=blobs
-METRICS_HTTP_PORT=7472
+METRICS_HTTP_PORT=7472

packages/blobstorage/.default.server.env

@@ -1,7 +1,7 @@
 HTTP_PORT=3282
 DATABASE_HOST=localhost
-DATABASE_PORT=27017
+DATABASE_PORT=3306
 DATABASE_USER=root
-DATABASE_PASSWORD=root
+DATABASE_PASSWORD=passw00d
 DATABASE_NAME=blobs
-METRICS_HTTP_PORT=7471
+METRICS_HTTP_PORT=7471

packages/blobstorage/README.md

@@ -1,55 +1,93 @@
 # blob-storage
 
-Repo for BLOB storage (archive and serve data)
+Repository for BLOB storage (archive and serve data)
 
-## how to run ?
+## Prerequisites
 
-Prerequisite is to have docker engine up and running.
+- Docker engine up and running.
+- Go installed on your system.
 
-1. Start the mongoDB
+## Configuration and Setup
 
-```bash
-cd local_docker && docker-compose up -d
-```
+### Setting up MySQL
 
-2. Start the `indexer`
+1. **Start MySQL**:
 
-```bash
-ENV_FILE=.default.indexer.env run cmd/main.go indexer
-```
+   Navigate to the `docker-compose` directory and start the MySQL service:
 
-By default the above command starts the app from the latest block height. If we want to specifiy a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable.
+   ```bash
+   cd ./docker-compose
+   docker-compose up -d
+   ```
 
-2. Start the `server`.
+   This command starts your MySQL instance as defined in your `docker-compose.yml` file.
 
-```bash
-ENV_FILE=.default.server.env run cmd/main.go server
-```
+2. **Migrate Database Schema**:
 
-## how to test / use ?
+   Navigate to the `migrations` directory to apply database migrations:
 
-When the `DB`, `blob-catcher` and `server` is running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, tho it is not written into the DB). Use that `blobHash` (including the 0x) in
+   ```bash
+   cd ./migrations
+   goose mysql "root:passw00d@tcp(localhost:3306)/blobs" status
+   goose mysql "root:passw00d@tcp(localhost:3306)/blobs" up
+   ```
 
-1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array):
+   These commands apply migrations to the `blobs` database.
 
-```bash
-curl -X GET "http://localhost:27001/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH"
-```
+### Environment Configuration
 
-The result will be something like this:
+Ensure your `.default.indexer.env` and `.default.server.env` files are configured with the correct database credentials, host, and any other necessary environment variables.
 
-```bash
-{"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]}
-```
+## Running the Application
 
-2. Or to backtest, use the simple python script below, after overwriting the `blob_hash` variable:
+1. **Start the Indexer**:
 
-```bash
-python3 python_query.py
-```
+   With the environment file configured, start the indexer:
 
-## todos
+   ```bash
+   ENV_FILE=.default.indexer.env go run cmd/main.go indexer
+   ```
+
+   This starts the app from the latest block height by default. Adjust the `STARTING_BLOCK_ID` in the environment file if needed.
+
+2. **Start the Server**:
+
+   Similarly, start the server:
+
+   ```bash
+   ENV_FILE=.default.server.env go run cmd/main.go server
+   ```
+
+## Testing and Usage
+
+When the `DB`, `blob-catcher` and `server` are running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, though it is not written into the DB). Use that `blobHash` (including the 0x) in
+
+1. **Querying Blob Data via HTTP Request**:
+
+   To retrieve blob data, you can execute a `curl` command. This allows for querying multiple `blobHashes` simultaneously, separated by commas. A single request can yield an array of results:
+
+   ```bash
+   curl -X GET "http://localhost:3282/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH"
+   ```
+
+   **Expected Output**:
+
+   ```bash
+   {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]}
+   ```
+
+2. **Backtesting with a Python Script**:
+
+   This script facilitates querying the database directly based on a specified `blob_hash`. Modify the `blob_hash` variable in the script to match the hash you wish to query.
+
+   To run the script:
+
+   ```bash
+   python3 python_query.py
+   ```
+
+## Todos
 
 What is still missing is:
 
-- small refinements and DevOps (prod-grade DB with creditentials, proper containerization)
+- small refinements and DevOps (prod-grade DB with credentials, proper containerization)

packages/blobstorage/docker-compose/docker-compose.yml

@@ -0,0 +1,14 @@
+version: "3.8"
+services:
+  db:
+    image: mysql:8.0
+    cap_add:
+      - SYS_NICE
+    restart: always
+    environment:
+      - MYSQL_DATABASE=blobs
+      - MYSQL_ROOT_PASSWORD=passw00d
+    ports:
+      - "3306:3306"
+    volumes:
+      - ~/.docker-conf/mysql/data/:/var/lib/mysql

packages/blobstorage/python_query.py

@@ -1,20 +1,27 @@
-from pymongo import MongoClient
+import mysql.connector
 
 def query_blob_hash(collection_name, blob_hash):
-    # Connect to MongoDB
-    client = MongoClient('localhost', 27017)
 
-    # Access the database
-    db = client['blob_storage']
+    # Connect to MySQL
+    connection = mysql.connector.connect(
+        host='localhost',
+        user='root',  # Update with your username
+        password='passw00d',  # Update with your password
+        database='blobs'  # Update with your database name
+    )
 
-    # Access the collection
-    collection = db[collection_name]
+    cursor = connection.cursor(dictionary=True)
 
-    # Query the collection for the blobHash
-    result = collection.find_one({'blob_hash': blob_hash})
+    # Query the table for the blobHash
+    query = f"SELECT * FROM blob_hashes WHERE blob_hash = %s"
+    cursor.execute(query, (blob_hash,))
 
-    # Close the connection
-    client.close()
+    # Fetch one result
+    result = cursor.fetchone()
+
+    # Close the cursor and connection
+    cursor.close()
+    connection.close()
 
     return result