Skip to content

Development Readme

Jump to bottom
user_name edited this page Aug 7, 2023 · 13 revisions

Development Overview

IMPORTANT NOTE TO THE READER: This document aims to be the "get me going document", so it is unlikely here will still be a lot of things and tools you do not know after finishing it. However, it is intended for you to go through from start to finish in order to lay a foundation with regard to everything you need to know. PLEASE update it with additional resources you find helpful along the way.

TODO(@Olshansk, @bryanchriswhite): Make sure to update this document after #730 and #874 are merged in.

LFG - Development

Install Dependencies

  • Install Docker
  • Install Docker Compose
  • Install Golang
  • protoc-gen-go, protoc-go-inject-tag and mockgen by running make install_cli_deps

Note to the reader: Please update this list if you found anything missing.

Last tested by with:

docker --version
# Docker version 20.10.14, build a224086

protoc --version
# libprotoc 3.19.4

which protoc-go-inject-tag && echo "protoc-go-inject-tag Installed"

# /your$HOME/go/bin/protoc-go-inject-tag
# protoc-go-inject-tag Installed

go version
# go version go1.20.5 darwin/arm64

mockgen --version
# v1.6.0

system_profiler SPSoftwareDataType
# Software:
#
#     System Software Overview:
#
#       System Version: macOS 12.3.1 (21E258)
#       Kernel Version: Darwin 21.4.0

Prepare Local Environment

Generate local files

git clone [email protected]:pokt-network/pocket.git && cd pocket
make develop_start

Optionally activate changelog pre-commit hook

cp .githooks/pre-commit .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

NOTE: The pre-commit changelog verification has been disabled during the developement of V1 as of 2023-05-16 to unblock development velocity; see more details here. This check is no longer done in the CI and is not recommended for local development either currently.

Pocket Network CLI

The Pocket node provides a CLI for interacting with Pocket RPC Server. The CLI can be used for both read & write operations by users and to aid in automation.

In order to build the CLI:

  1. Generate local files
make develop_start
  1. Build the CLI binary
make build

The cli binary will be available at bin/p1 and can be used instead of go run app/client/*.go

The commands available are listed here or accessible via bin/p1 --help

2.1 [OPTIONAL] Add the binary to your .rc

IMPORTANT: Note that this helper ONLY works with the docker-compose LocalNet setup and DOES NOT work with the k8s setup TODO: This section can be deleted once the CLI accepts a --remoteURL option.

You can add the following function so you can run the p1 from anywhere on your host.

function p1 {
    export POCKET_WORKDIR="${HOME}/workspace/pocket/pocket/"
    export CONFIG_PATH="${POCKET_WORKDIR}/build/config/config.validator1.json"
    export GENESIS_PATH="${POCKET_WORKDIR}/build/config/genesis_localhost.json"
    ${POCKET_WORKDIR}/bin/p1 "$@"
}

Execute the following command from the root of the project to add it to your ~/.bash_aliases or similar.

cat >> ~/.bash_aliases << FUNC
function p1 {
    export POCKET_WORKDIR="$(pwd)"
    export CONFIG_PATH="${POCKET_WORKDIR}/build/config/config.validator1.json"
    export GENESIS_PATH="${POCKET_WORKDIR}/build/config/genesis_localhost.json"
    ${POCKET_WORKDIR}/bin/p1 "\$@"
}
FUNC

You can via a demo of it here.

Swagger UI

Swagger UI is available to help during the development process.

In order to spin a local instance of it with the API definition for the Pocket Network Node RPC interface automatically pre-loaded you can run:

make swagger-ui

View Available Commands

make

Running Unit Tests

make test_all

Note that there are a few tests in the library that are prone to race conditions and we are working on improving them. This can be checked with make test_race.

Running LocalNet

At the time of writing, we have two basic approaches to running a LocalNet. We suggest getting started with the Docker Compose (aka lightweight LocalNet) approach outlined below before moving to the advanced Kubernetes (aka LocalNet) configuration.

[Advanced] Kubernetes

The full documentation on running a LocalNet on kubernetes can be found here.

[Basic] Docker Compose

V1 Localnet Demo

  1. Delete any previous docker state
make docker_wipe
  1. In one shell, run the 4 nodes setup:
make lightweight_localnet
  1. In another shell, run the development client:
make lightweight_localnet_client && make lightweight_localnet_client_debug
  1. Check the state of each node:
✔ PrintNodeState
  1. Trigger the next view to ensure everything is working:
✔ TriggerNextView
  1. Reset the ResetToGenesis if you want to:
✔ ResetToGenesis
  1. Set the client to automatic and watch it go:
✔ TogglePacemakerMode
✔ TriggerNextView
  1. [Optional] Common manual set of verification steps
✔ ResetToGenesis
✔ PrintNodeState # Check committed height is 0
✔ TriggerNextView
✔ PrintNodeState # Check committed height is 1
✔ TriggerNextView
✔ PrintNodeState # Check committed height is 2
✔ TogglePacemakerMode # Check that it’s automatic now
✔ TriggerNextView # Let it rip!
  1. Send a transaction
  make send_local_tx
  1. Trigger the next view
✔ TriggerNextView # Commit the transaction

Build a container image from development's branch

tl;dr Attach the push-image label to the PR to build and push an image from your branch

If you need to test or run non-merged pocket code outside LocalNet, for example on DevNet, TestNet or somewhere else, you can force the CI to build and push a container image. This can be done by attaching a push-image label on your Pull Request. Next CI build will push the container image to our container registry.

TODO: Improvements to be added by the core team

A lot of features have been added since this doc was first added. See docs/demo. We should update it to:

  1. Show k8s LocalNet
  2. Add more details related to transactions
  3. Add details related to the keybase
  4. Add state sync tooling
  5. Add P2P tooling

Profiling

If you need to profile the node for CPU and/or memory usage, you can use the pprof tool. A quick guide is available here.

Code Organization

Pocket
├── app                               # Entrypoint to running the Pocket node and clients
│   ├── client                        # Entrypoint to running a local Pocket debug client
│   └── pocket                        # Entrypoint to running a local Pocket node
├── bin                               # Destination for compiled pocket binaries
├── build                             # Build related source files including Docker, scripts, etc
│   ├── config                        # Configuration files for to run nodes in development
│   ├── deployments                   # Docker-compose to run different cluster of services for development
│   ├── docs                          # Links to V1 Protocol implementation documentation (excluding the protocol specification)
├── consensus                         # Implementation of the Consensus module
├── docs                              # Links to V1 Protocol implementation documentation (excluding the protocol specification)
├── internal                          # Internal package following the convention established in Go 1.4: https://go.dev/doc/go1.4#internalpackages
│   └── testutil                      # Internal package for reusable and/or common test code
├── logger                            # Implementation of the Logger module
├── p2p                               # Implementation of the P2P module
├── persistence                       # Implementation of the Persistence module
├── rpc                               # Implementation of the RPC module
├── runtime                           # Implementation of the Runtime module
│   ├── configs                       # Configuration struct definitions
│   │   └── proto                     # Protobuf representing the specific configuration of the various modules
│   ├── defaults                      # Default values for the configuration structs
│   ├── genesis
│   │   └── proto                     # Protobuf representing the genesis state of the Pocket blockchain
│   └── test_artifacts                # Componentry used for generating test artifacts such as particular genesis states used in testing
├── shared                            # Shared types, modules and utils
│   ├── codec
│   │   └── proto
│   ├── converters
│   ├── core                          # Core types (Actor, Pools, etc.) used throughout the codebase
│   │   └── types
│   │       └── proto                 # Protobuf representing the core types used throughout the codebase
│   ├── crypto
│   ├── docs
│   │   └── flows
│   ├── messaging                     # Messaging structs and functions
│   │   └── proto
│   └── modules                       # Shared modules definitions (interfaces)
├── telemetry                         # Implementation of the Telemetry module
├── utility                           # Implementation of the Utility module
└── Makefile                          # The source of targets used to develop, build and test

Maintaining Documentation

Documentation files currently found by the following command find . -name "*.md" | grep -v -e "vendor" -e "app" are added to the Github Wiki Repository. The Wiki will be improved overtime but in its current form, provides an organized overview of the Repository.

To keep the Wiki organized, a comment is added at the end of each .md file. For example, you can find the following one at the end of this file <!-- GITHUB_WIKI: guides/development/readme -->. The structure of the comment indicates the category (guides), subcategory(ies) (development) and filename (readme): <!-- GITHUB_WIKI: <category>/<subcategory 1>/.../<filename>. You can see the example output in the Wiki Page.

If you are adding a new .md file for documentation please included a similar comment. Use your best judgment for the category and subcategory if its a new directory. Otherwise, copy the comment from a similar file in the directory and choose a relevant filename.

Code Review Guidelines

Documentation Resources and Implementation

Your Project Dashboard

If you plan to be a regular contributor, you might benefit from having your own dashboard similar to this to help track your work. Reach out to one of the maintainers to help you set it up.

Github Labels

We have various labels on GitHub for filtering and tagging different tickets, but some of them trigger special logic.

  • cl validate - shares the comments related to changelog validation as a PR comment; merging is still blocked by default
  • gpt review - adds a code review by GPT to your PR
  • do not merge - blocks merging the PR even if it is approved and CI is green
  • community - is automatically added to Pocket's dashboard on app.dework.xyz so it becomes a bounty for a community contributor
  • push-image - builds and pushes an image at the HEAD of the PR
  • e2e-devnet-test - runs E2E tests on your PR
  • Not a label but close: Add the /reviewpad summarize comment to automatically get an LLM to summarize your PR

Github Actions File

The Action is triggered when there is a change to any Markdown file on the main branch of the Repository. When triggered, environment variables are set for a Python script that updates the Github Wiki Repository based on Pocket Repository files.

Python Script

The script finds the relevant Markdown files in the repository and organizes them for the Wiki Repository. Currently, the find command is filtered to exclude the ./app and ./vendor directories. Based on the list of .md file paths, it maps the formatting spec from above to some information about the file. Using the map, it creates a Sidebar file format which Github uses as a Table of Contents for the wiki. Also, from the Pocket repo we copy over the files with titles linking to the Sidebar format.

Below, you can see some of the patterns between the Sidebar format, folder of markdowns in the Wiki Repository, and final sidebar/table of contents display.

Format Folder Wiki
format Folder wiki

Linters

We utilize golangci-lint to run the linters. It is a wrapper around a number of linters and is configured to run many at once. The linters are configured to run on every commit and pull request via CI, and all code issues are populated as GitHub annotations to let developers and reviewers easily locate an issue.

Installation of golangci-lint

Please follow the instructions on the golangci-lint website.

Running linters locally

You can run golangci-lint locally against all packages with:

make go_lint

If you need to specify any additional flags, you can run golangci-lint directly as it picks up the configuration from the .golangci.yml file.

VSCode Integration

golangci-lint has an integration with VSCode. Per documentation, recommended settings are:

"go.lintTool":"golangci-lint",
"go.lintFlags": [
  "--fast"
]

Configuration

golangci-lint is a sophisticated tool including both default and custom linters. The configuration file, which can grow quite large, is located at .golangci.yml.

The official documentation includes a list of different linters and their configuration options. Please refer to this page for more information.

Custom linters

We can write custom linters using go-ruleguard. The rules are located in the build/linters directory. The rules are written in the Ruleguard DSL, if you've never worked with ruleguard in the past, it makes sense to go through introduction article and Ruleguard by example tour.

Ruleguard is run via gocritic linter which is a part of golangci-lint, so if you wish to change configuration or debug a particular rule, you can modify the .golangci.yml file.

Contents

Clone this wiki locally