Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
chore(repo): add blob-catcher to monorepo (#16353)
Co-authored-by: maskpp <[email protected]> Co-authored-by: jeff <[email protected]> Co-authored-by: Jeffery Walsh <[email protected]>
- Loading branch information
4 people
authored
Mar 14, 2024
1 parent
07a1a81
commit ecbf797
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| name: Blobstorage | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| paths: | ||
| - "packages/blobstorage/**" | ||
| pull_request: | ||
| paths: | ||
This comment has been minimized.
Sorry, something went wrong. |
||
| - "packages/blobstorage/**" | ||
|
|
||
| jobs: | ||
| test: | ||
| runs-on: [taiko-runner] | ||
| steps: | ||
| - name: Cancel Previous Runs | ||
| uses: styfle/[email protected] | ||
| with: | ||
| access_token: ${{ github.token }} | ||
|
|
||
| - uses: actions/checkout@v4 | ||
| - uses: actions/setup-go@v5 | ||
| with: | ||
| go-version: "1.21.0" | ||
|
|
||
| - name: blobstorage - Unit Tests | ||
| working-directory: ./packages/blobstorage | ||
| run: go test `go list ./... | grep -v ./bindings | grep -v ./mock | grep -v ./cmd` -coverprofile=coverage.txt -covermode=atomic | ||
|
|
||
| - name: blobstorage - Upload coverage to Codecov | ||
| uses: codecov/codecov-action@v4 | ||
| with: | ||
| files: ./packages/blobstorage/coverage.txt | ||
| flags: blobstorage | ||
|
|
||
| push-docker-image: | ||
| # only push docker image on PR merge to main | ||
| if: ${{ github.event }} == 'push' | ||
| name: Build and push docker image | ||
| runs-on: [taiko-runner] | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| ref: ${{ github.ref }} | ||
|
|
||
| - name: Login to GCR | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| registry: gcr.io | ||
| username: _json_key | ||
| password: ${{ secrets.GCR_JSON_KEY }} | ||
|
|
||
| - name: Set up QEMU | ||
| uses: docker/setup-qemu-action@v3 | ||
|
|
||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
|
|
||
| - name: Docker meta | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
This comment has been minimized.
Sorry, something went wrong. |
||
| with: | ||
| images: | | ||
| gcr.io/evmchain/blobstorage | ||
| tags: | | ||
| type=ref,event=branch | ||
| type=ref,event=pr | ||
| type=ref,event=tag | ||
| type=sha | ||
| - name: Build and push | ||
| uses: docker/build-push-action@v5 | ||
| with: | ||
| platforms: linux/amd64 | ||
| push: true | ||
| context: . | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
| labels: ${{ steps.meta.outputs.labels }} | ||
| build-args: | | ||
| PACKAGE=blobstorage | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| RPC_URL=wss://l1ws.internal.taiko.xyz | ||
| BEACON_URL=https://l1beacon.internal.taiko.xyz/eth/v1/beacon/blob_sidecars/ | ||
| TAIKO_L1_CONTRACT_ADDRESS=0xC069c3d2a9f2479F559AD34485698ad5199C555f | ||
| DATABASE_HOST=localhost | ||
| DATABASE_PORT=27017 | ||
| DATABASE_USER=root | ||
| DATABASE_PASSWORD=root | ||
| DATABASE_NAME=blobs | ||
| METRICS_HTTP_PORT=7472 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| PORT=3282 | ||
| DATABASE_HOST=localhost | ||
| DATABASE_PORT=27017 | ||
| DATABASE_USER=root | ||
| DATABASE_PASSWORD=root | ||
| DATABASE_NAME=blobs | ||
| METRICS_HTTP_PORT=7471 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # If you prefer the allow list template instead of the deny list, see community template: | ||
| # https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore | ||
| # | ||
| # Binaries for programs and plugins | ||
| *.exe | ||
| *.exe~ | ||
| *.dll | ||
| *.so | ||
| *.dylib | ||
|
|
||
| # Test binary, built with `go test -c` | ||
| *.test | ||
|
|
||
| # Output of the go coverage tool, specifically when used with LiteIDE | ||
| *.out | ||
|
|
||
| # Dependency directories (remove the comment below to include it) | ||
| # vendor/ | ||
|
|
||
| # Go workspace file | ||
| go.work |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # blob-storage | ||
|
|
||
| Repo for BLOB storage (archive and serve data) | ||
|
|
||
| ## how to run ? | ||
|
|
||
| Prerequisite is to have docker engine up and running. | ||
|
|
||
| 1. Start the mongoDB | ||
|
|
||
| ```bash | ||
| cd local_docker && docker-compose up -d | ||
| ``` | ||
|
|
||
| 2. Start the `indexer` | ||
|
|
||
| ```bash | ||
| ENV_FILE=.default.indexer.env run cmd/main.go indexer | ||
| ``` | ||
|
|
||
| By default the above command starts the app from the latest block height. If we want to specifiy a previous blockheight, we can run change it from the `.default.indexer.env` file, by adding a `STARTING_BLOCK_ID` variable. | ||
|
|
||
| 2. Start the `server`. | ||
|
|
||
| ```bash | ||
| ENV_FILE=.default.server.env run cmd/main.go server | ||
| ``` | ||
|
|
||
| ## how to test / use ? | ||
|
|
||
| When the `DB`, `blob-catcher` and `server` is running, the `blob-catcher` is outputting the `blobHash` to the terminal (with the `networkName` variable too, tho it is not written into the DB). Use that `blobHash` (including the 0x) in | ||
|
|
||
| 1. Either in a curl command like this (you can query multiple blobHashes - comma separated - with one go and the result will be a respective array): | ||
|
|
||
| ```bash | ||
| curl -X GET "http://localhost:27001/getBlob?blobHash=0x01a2a1cdc7ad221934061642a79a760776a013d0e6fa1a1c6b642ace009c372a,0xWRONG_HASH" | ||
| ``` | ||
|
|
||
| The result will be something like this: | ||
|
|
||
| ```bash | ||
| {"data":[{"blob":"0x123...00","kzg_commitment":"0xabd68b406920aa74b83cf19655f1179d373b5a8cba21b126b2c18baf2096c8eb9ab7116a89b375546a3c30038485939e"}, {"blob":"NOT_FOUND","kzg_commitment":"NOT_FOUND"}]} | ||
| ``` | ||
|
|
||
| 2. Or to backtest, use the simple python script below, after overwriting the `blob_hash` variable: | ||
|
|
||
| ```bash | ||
| python3 python_query.py | ||
| ``` | ||
|
|
||
| ## todos | ||
|
|
||
| What is still missing is: | ||
|
|
||
| - small refinements and DevOps (prod-grade DB with creditentials, proper containerization) |
Oops, something went wrong.
its good