diff --git a/Makefile b/Makefile index 03ed907..807551f 100644 --- a/Makefile +++ b/Makefile @@ -40,6 +40,7 @@ help: @echo " fmt - Format code" @echo " clippy - Run clippy lints" @echo " clean - Clean build artifacts" + @echo " book - Build the user documentation (mdbook)" @echo "" @echo "Environment Variables:" @echo "" @@ -329,6 +330,10 @@ clippy: clean: $(CARGO) clean +.PHONY: book +book: + mdbook build book + # Dafny commands dafny-translate: @echo "Translating Dafny specification to Rust..." diff --git a/book/book.toml b/book/book.toml new file mode 100644 index 0000000..1ecfb0f --- /dev/null +++ b/book/book.toml @@ -0,0 +1,11 @@ +[book] +authors = ["Rayls Network"] +language = "en" +src = "src" +title = "RBFT User Guide" + +[build] +build-dir = "../target/book" + +[output.html] +git-repository-url = "https://github.com/raylsnetwork/rbft" diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md new file mode 100644 index 0000000..5ccf5b5 --- /dev/null +++ b/book/src/SUMMARY.md @@ -0,0 +1,39 @@ +# Summary + +[Introduction](./introduction.md) + +# Getting Started + +- [Prerequisites](./getting-started/prerequisites.md) +- [Installation](./getting-started/installation.md) + +# Running a Chain + +- [Local Testnet](./running/local-testnet.md) +- [Follower Nodes](./running/follower-nodes.md) +- [Validator Management](./running/validator-management.md) +- [Docker Deployment](./running/docker.md) +- [Kubernetes Deployment](./running/kubernetes.md) + +# Deploying Contracts + +- [Using Cast (Foundry)](./contracts/cast.md) +- [QBFTValidatorSet Contract](./contracts/validator-set.md) +- [Deploying Your Own Contracts](./contracts/deploying.md) + +# Interacting with the Chain + +- [Sending Transactions](./interacting/transactions.md) +- [ERC20 Tokens](./interacting/erc20.md) +- [Load Testing](./interacting/load-testing.md) + +# Operations + +- [Configuration Reference](./operations/configuration.md) +- [Monitoring and Logs](./operations/monitoring.md) +- [Troubleshooting](./operations/troubleshooting.md) + +# Architecture + +- [Overview](./architecture/overview.md) +- [QBFT Consensus](./architecture/qbft.md) diff --git a/book/src/architecture/overview.md b/book/src/architecture/overview.md new file mode 100644 index 0000000..9038c4e --- /dev/null +++ b/book/src/architecture/overview.md @@ -0,0 +1,89 @@ +# Architecture Overview + +RBFT is built as a layered system on top of Reth, the Rust Ethereum client. + +``` +┌─────────────────┐ +│ RPC Interface │ HTTP JSON-RPC (ports 8545+) +└────────┬────────┘ + │ +┌────────▼────────┐ +│ Reth Engine │ EVM execution, state management, transaction pool +└────────┬────────┘ + │ +┌────────▼────────┐ +│ RBFT Consensus │ QBFT protocol, proposer rotation, validator management +└────────┬────────┘ + │ +┌────────▼────────┐ +│ RLPx Network │ P2P messaging between validators and followers +└─────────────────┘ +``` + +## Crate structure + +### `rbft` (core library) + +The consensus protocol implementation with no Reth dependencies. Contains: + +- QBFT state machine (`NodeState`) +- Message types (Proposal, Prepare, Commit, RoundChange, NewBlock) +- Block validation and proposer election +- Round change and timeout logic + +This crate is designed to be testable in isolation using an in-memory +`NodeSwarm` simulation. + +### `rbft-node` (binary) + +Integrates the core library with Reth: + +- Custom consensus engine (`RbftConsensus`) +- RLPx protocol handler for QBFT messages +- Block building and execution via Reth +- P2P connection management + +### `rbft-utils` (binary) + +Operator tooling: + +- Genesis generation (compiles and deploys QBFTValidatorSet contract) +- Testnet orchestration (start/stop/monitor nodes) +- Validator management (add/remove/status via contract calls) +- Log analysis (logjam) + +### `rbft-megatx` (binary) + +Transaction load generator for stress testing. + +### `rbft-validator-inspector` (binary) + +Terminal UI for real-time validator monitoring. + +## Node types + +| Type | Has validator key | Participates in consensus | Produces blocks | +|---|---|---|---| +| Validator (proposer) | Yes | Yes | Yes (when elected) | +| Validator (non-proposer) | Yes | Yes | No | +| Follower | No | No | No | + +## Data flow + +1. Transactions arrive via JSON-RPC +2. The current proposer builds a block and broadcasts a `Proposal` +3. Validators verify and send `Prepare` messages +4. On quorum of prepares, validators send `Commit` messages +5. On quorum of commits, the block is finalized +6. A `NewBlock` message is broadcast so followers can update + +## Fault tolerance + +With `n` validators, RBFT tolerates `f = (n-1) / 3` Byzantine faults. +The quorum size is `(2n - 1) / 3 + 1`. + +| Validators | Max faults | Quorum | +|---|---|---| +| 4 | 1 | 3 | +| 7 | 2 | 5 | +| 10 | 3 | 7 | diff --git a/book/src/architecture/qbft.md b/book/src/architecture/qbft.md new file mode 100644 index 0000000..eb781a7 --- /dev/null +++ b/book/src/architecture/qbft.md @@ -0,0 +1,83 @@ +# QBFT Consensus + +RBFT implements the QBFT (Quorum Byzantine Fault Tolerant) consensus protocol, +a practical BFT algorithm designed for permissioned blockchain networks. + +## References + +- [The Istanbul BFT Consensus Algorithm (IBFT paper)](https://arxiv.org/abs/2002.03613) +- [QBFT Formal Specification (ConsenSys)](https://github.com/ConsenSys/qbft-formal-spec-and-verification) + +## Protocol overview + +QBFT operates in rounds within each block height. Each round has a designated +proposer determined by round-robin rotation. + +### Normal case (round 0) + +1. **Block timeout** — the proposer waits for the block interval, then + broadcasts a `Proposal` containing the new block. +2. **Prepare** — validators verify the proposal and broadcast a `Prepare` + message. +3. **Commit** — once a validator receives a quorum of prepares, it broadcasts + a `Commit` message with a commit seal. +4. **Finalize** — once a quorum of commits is collected, the block is added + to the chain with the commit seals embedded in the header. +5. **NewBlock** — the committed block is broadcast so all nodes (including + followers) can update their chains. + +### Round change + +If the proposer fails to propose within the timeout, validators trigger a +round change: + +1. Each validator sends a `RoundChange` message for the next round. +2. When a quorum of round changes is collected, the new round's proposer + creates a proposal with a justification (the round change messages). +3. The protocol continues from step 2 (prepare phase). + +### Timeout schedule + +Round timeouts grow exponentially to avoid thrashing: + +``` +timeout(r) = first_interval * growth_factor^r +``` + +Default values: + +| Parameter | Default | +|---|---| +| `first_interval` | 1.0 s | +| `growth_factor` | 2.0 | +| `max_round` | 10 | + +## Proposer election + +The proposer for round 0 at a given height is the validator after the previous +block's proposer (wrapping around). For subsequent rounds, the index advances +by the round number: + +``` +round_zero_index = index_of(previous_proposer) + 1 +proposer_index = (round_zero_index + round) % num_validators +``` + +At height 1, the first validator (index 0) proposes. + +## Quorum and fault tolerance + +For `n` validators: + +- Maximum Byzantine faults tolerated: `f = (n-1) / 3` +- Quorum size: `q = (2n - 1) / 3 + 1` + +The protocol guarantees safety (no conflicting blocks are finalized) as long as +at most `f` validators are faulty. It guarantees liveness as long as at least +`q` validators are honest and connected. + +## Validator sets and epochs + +The validator set can change dynamically through the QBFTValidatorSet contract. +Changes take effect at epoch boundaries (every `epochLength` blocks) to ensure +all validators agree on the set for a given block range. diff --git a/book/src/contracts/cast.md b/book/src/contracts/cast.md new file mode 100644 index 0000000..0fd07ab --- /dev/null +++ b/book/src/contracts/cast.md @@ -0,0 +1,67 @@ +# Using Cast (Foundry) + +[Cast](https://book.getfoundry.sh/cast/) is a CLI tool from Foundry for +interacting with EVM-compatible chains. It is the recommended way to send +transactions and query state on an RBFT network. + +## Installation + +```bash +curl -L https://foundry.paradigm.xyz | bash +foundryup +``` + +Verify: + +```bash +cast --version +``` + +## Common operations + +All examples assume the default testnet RPC at `http://localhost:8545`. + +### Check block height + +```bash +cast bn +``` + +### Query an account balance + +```bash +cast balance 0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf +``` + +### Send ETH + +```bash +cast send 0x1234567890123456789012345678901234567890 \ + --value 1ether \ + --private-key 0x \ + --rpc-url http://localhost:8545 +``` + +### Derive an address from a private key + +```bash +cast wallet address --private-key 0x +``` + +### Call a contract (read-only) + +```bash +cast call 0x0000000000000000000000000000000000001001 \ + "getValidators()(address[])" \ + --rpc-url http://localhost:8545 +``` + +### Send a contract transaction + +```bash +cast send 0x0000000000000000000000000000000000001001 \ + "addValidator(address,string)" \ + 0xABCD... "enode://..." \ + --private-key 0x \ + --rpc-url http://localhost:8545 +``` diff --git a/book/src/contracts/deploying.md b/book/src/contracts/deploying.md new file mode 100644 index 0000000..4b75311 --- /dev/null +++ b/book/src/contracts/deploying.md @@ -0,0 +1,68 @@ +# Deploying Your Own Contracts + +RBFT chains are fully EVM-compatible. Any contract that works on Ethereum can be +deployed to an RBFT network using standard tools. + +## Using Foundry (forge) + +Create a project and deploy: + +```bash +forge init my-contract +cd my-contract + +# Edit src/Counter.sol, then deploy: +forge create src/Counter.sol:Counter \ + --rpc-url http://localhost:8545 \ + --private-key 0x +``` + +## Using cast with raw bytecode + +```bash +cast send --create \ + --private-key 0x \ + --rpc-url http://localhost:8545 +``` + +## Using Hardhat + +In `hardhat.config.js`: + +```javascript +module.exports = { + networks: { + rbft: { + url: "http://localhost:8545", + accounts: ["0x"] + } + } +}; +``` + +Deploy: + +```bash +npx hardhat run scripts/deploy.js --network rbft +``` + +## Getting test funds + +On a default local testnet, the admin account +(`0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf`, derived from key `0x000...0001`) +is pre-funded with ETH. Transfer from it to fund your deployment account: + +```bash +cast send 0x \ + --value 100ether \ + --private-key 0x0000000000000000000000000000000000000000000000000000000000000001 \ + --rpc-url http://localhost:8545 +``` + +## Chain ID + +The chain ID is set in `genesis.json`. Check it with: + +```bash +cast chain-id --rpc-url http://localhost:8545 +``` diff --git a/book/src/contracts/validator-set.md b/book/src/contracts/validator-set.md new file mode 100644 index 0000000..3871780 --- /dev/null +++ b/book/src/contracts/validator-set.md @@ -0,0 +1,71 @@ +# QBFTValidatorSet Contract + +The QBFTValidatorSet contract manages the validator set on-chain. It is deployed +at genesis and lives at a fixed address. + +## Contract details + +| Property | Value | +|---|---| +| Address | `0x0000000000000000000000000000000000001001` | +| Pattern | UUPS upgradeable proxy | +| Source | `contracts/QBFTValidatorSet.sol` | + +## Roles + +| Role | Permissions | +|---|---| +| `DEFAULT_ADMIN_ROLE` | Grant/revoke roles, upgrade contract | +| `VALIDATOR_MANAGER_ROLE` | Add/remove validators, update parameters | +| `UPGRADER_ROLE` | Upgrade contract implementation | + +The admin key used at genesis holds all roles by default. + +## Key functions + +### Read-only + +| Function | Description | +|---|---| +| `getValidators()` | Returns the current validator address list | +| `isValidator(address)` | Checks if an address is a validator | +| `maxActiveValidators()` | Returns the maximum validator count | +| `baseFee()` | Returns the current base fee | +| `blockIntervalMs()` | Returns the block interval in milliseconds | +| `epochLength()` | Returns the epoch length in blocks | + +### State-changing (admin only) + +| Function | Description | +|---|---| +| `addValidator(address, enode)` | Add a validator | +| `removeValidator(address)` | Remove a validator | +| `setMaxActiveValidators(uint256)` | Update max validators | +| `setBaseFee(uint256)` | Update base fee | +| `setBlockIntervalMs(uint256)` | Update block interval | +| `setEpochLength(uint256)` | Update epoch length | + +## Epochs + +Validator set changes are batched per epoch. When a validator is added or +removed, the change is recorded immediately but only takes effect when the +current epoch ends (every `epochLength` blocks). This prevents validator +churn from disrupting consensus mid-epoch. + +## Querying with rbft-utils + +```bash +target/release/rbft-utils validator status --rpc-url http://localhost:8545 +``` + +## Querying with cast + +```bash +# List validators +cast call 0x0000000000000000000000000000000000001001 \ + "getValidators()(address[])" + +# Check if an address is a validator +cast call 0x0000000000000000000000000000000000001001 \ + "isValidator(address)(bool)" 0xABCD... +``` diff --git a/book/src/getting-started/installation.md b/book/src/getting-started/installation.md new file mode 100644 index 0000000..0339700 --- /dev/null +++ b/book/src/getting-started/installation.md @@ -0,0 +1,24 @@ +# Installation + +Clone the repository and build in release mode: + +```bash +git clone https://github.com/raylsnetwork/rbft.git +cd rbft +cargo build --release +``` + +This produces three main binaries in `target/release/`: + +| Binary | Purpose | +|---|---| +| `rbft-node` | Run a validator or follower node | +| `rbft-utils` | Genesis generation, testnet management, validator operations | +| `rbft-megatx` | Transaction load testing | + +Verify the build: + +```bash +target/release/rbft-node --version +target/release/rbft-utils --help +``` diff --git a/book/src/getting-started/prerequisites.md b/book/src/getting-started/prerequisites.md new file mode 100644 index 0000000..6df5061 --- /dev/null +++ b/book/src/getting-started/prerequisites.md @@ -0,0 +1,22 @@ +# Prerequisites + +## Required + +- **Rust 1.93+** with nightly toolchain (for formatting) + ```bash + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh + rustup toolchain install nightly + ``` +- **Git** +- **Make** +- **Linux or macOS** (Windows via WSL) + +## Optional + +- **Foundry** — for interacting with deployed contracts via `cast` + ```bash + curl -L https://foundry.paradigm.xyz | bash + foundryup + ``` +- **Docker** — for containerized deployments +- **kubectl** — for Kubernetes deployments diff --git a/book/src/interacting/erc20.md b/book/src/interacting/erc20.md new file mode 100644 index 0000000..bb87159 --- /dev/null +++ b/book/src/interacting/erc20.md @@ -0,0 +1,39 @@ +# ERC20 Tokens + +RBFT chains are fully EVM-compatible, so any standard ERC20 token contract can +be deployed and used. + +## Deploying an ERC20 token + +Using Foundry with OpenZeppelin contracts: + +```bash +forge create lib/openzeppelin-contracts/contracts/token/ERC20/ERC20.sol:ERC20 \ + --constructor-args "MyToken" "MTK" \ + --rpc-url http://localhost:8545 \ + --private-key "$RBFT_ADMIN_KEY" +``` + +## Interacting with ERC20 contracts + +```bash +# Check balance +cast call \ + "balanceOf(address)(uint256)" 0x + +# Transfer tokens +cast send \ + "transfer(address,uint256)" 0x 1000000000000000000 \ + --private-key "$RBFT_ADMIN_KEY" \ + --rpc-url http://localhost:8545 + +# Check allowance +cast call \ + "allowance(address,address)(uint256)" 0x 0x + +# Approve spending +cast send \ + "approve(address,uint256)" 0x 1000000000000000000 \ + --private-key "$RBFT_ADMIN_KEY" \ + --rpc-url http://localhost:8545 +``` diff --git a/book/src/interacting/load-testing.md b/book/src/interacting/load-testing.md new file mode 100644 index 0000000..73d08ee --- /dev/null +++ b/book/src/interacting/load-testing.md @@ -0,0 +1,63 @@ +# Load Testing + +RBFT includes `rbft-megatx`, a tool for generating high transaction load to +stress-test the network. + +## Quick start + +```bash +make testnet_load_test +``` + +This starts a 4-validator testnet with `rbft-megatx` submitting large +transactions every block, monitoring the transaction pool. + +## Running megatx manually + +Against a running testnet: + +```bash +target/release/rbft-megatx spam \ + --num-txs 20000 \ + --target-tps 1000 \ + --max-wait-seconds 30 +``` + +| Flag | Default | Description | +|---|---|---| +| `--num-txs` | `20000` | Transactions per batch | +| `--target-tps` | `0` (unlimited) | Target transactions per second | +| `--max-wait-seconds` | `30` | Max wait for pending txs to clear | + +## Via the Makefile + +```bash +make megatx +``` + +Defaults to 100,000 transactions. + +## Monitoring the transaction pool + +When running with `--monitor-txpool`, the testnet command displays real-time +transaction pool sizes for each node, color-coded by congestion level. + +```bash +target/release/rbft-utils testnet --init --monitor-txpool --run-megatx \ + --assets-dir ~/.rbft/testnet/assets +``` + +## Tuning for high throughput + +For maximum throughput, increase pool limits: + +```bash +make testnet_load_test +``` + +This automatically sets: + +- `--txpool.max-tx-input-bytes 10000000` +- `--builder.gaslimit 60000000` +- `--tx-propagation-mode all` +- Large pending, basefee, and queued pool sizes diff --git a/book/src/interacting/transactions.md b/book/src/interacting/transactions.md new file mode 100644 index 0000000..e698c63 --- /dev/null +++ b/book/src/interacting/transactions.md @@ -0,0 +1,66 @@ +# Sending Transactions + +## Setup + +For local testing, use the pre-funded admin key: + +```bash +export RBFT_ADMIN_KEY=0x0000000000000000000000000000000000000000000000000000000000000001 +``` + +Derive the corresponding address: + +```bash +cast wallet address --private-key "$RBFT_ADMIN_KEY" +# 0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf +``` + +## Sending ETH + +Using cast: + +```bash +cast send 0x1234567890123456789012345678901234567890 \ + --value 1ether \ + --private-key "$RBFT_ADMIN_KEY" \ + --rpc-url http://localhost:8545 +``` + +Using curl: + +```bash +# First sign the transaction, then send the raw bytes via eth_sendRawTransaction +curl -X POST http://localhost:8545 \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "method": "eth_sendRawTransaction", + "params": ["0x"], + "id": 1 + }' +``` + +## Checking balances + +```bash +cast balance 0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf +``` + +Or with curl: + +```bash +curl -s -X POST http://localhost:8545 \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "method": "eth_getBalance", + "params": ["0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", "latest"], + "id": 1 + }' +``` + +## Transaction finality + +RBFT provides instant finality. Once a transaction is included in a committed +block, it is final — there are no reorgs or uncle blocks. A single block +confirmation is sufficient. diff --git a/book/src/introduction.md b/book/src/introduction.md new file mode 100644 index 0000000..365f389 --- /dev/null +++ b/book/src/introduction.md @@ -0,0 +1,37 @@ +# RBFT User Guide + +RBFT (Rayls Byzantine Fault Tolerant) is a high-performance BFT consensus +implementation based on QBFT, built on top of [Reth](https://github.com/paradigmxyz/reth). + +## What is RBFT? + +RBFT provides a permissioned Ethereum-compatible blockchain with fast finality. +Blocks are finalized as soon as they are committed by the validator quorum — there +are no forks or reorganizations under normal operation. + +Key features: + +- **QBFT Consensus** — standards-compliant QBFT with RLPx peer-to-peer networking +- **Reth Integration** — built on Reth for high-performance EVM execution +- **Dynamic Validators** — add or remove validators at runtime through smart contract interactions +- **Follower Nodes** — non-validator nodes that sync the chain without participating in consensus +- **Comprehensive Tooling** — genesis generation, testnet management, log analysis, and load testing + +## Who is this guide for? + +This guide is for operators and developers who want to: + +- Set up and run an RBFT chain +- Deploy and interact with smart contracts +- Manage validators and follower nodes +- Monitor and troubleshoot a running network + +## Project layout + +| Crate | Description | +|---|---| +| `rbft` | Core QBFT consensus protocol library | +| `rbft-node` | Node binary with Reth integration | +| `rbft-utils` | CLI for genesis generation, testnet management, and validator operations | +| `rbft-megatx` | Transaction load testing tool | +| `rbft-validator-inspector` | Terminal UI for monitoring validators | diff --git a/book/src/operations/configuration.md b/book/src/operations/configuration.md new file mode 100644 index 0000000..9e3ae05 --- /dev/null +++ b/book/src/operations/configuration.md @@ -0,0 +1,59 @@ +# Configuration Reference + +## Node flags (rbft-node) + +These flags are specific to RBFT. The node also accepts all standard +[Reth CLI flags](https://paradigmxyz.github.io/reth/cli/reth/). + +| Flag | Env Variable | Default | Description | +|---|---|---|---| +| `--validator-key` | `RBFT_VALIDATOR_KEY` | — | Validator private key file (omit for follower) | +| `--logs-dir` | `RBFT_LOGS_DIR` | `~/.rbft/testnet/logs` | Log directory | +| `--db-dir` | `RBFT_DB_DIR` | `~/.rbft/testnet/db` | Database directory | +| `--trusted-peers-refresh-secs` | `RBFT_TRUSTED_PEERS_REFRESH_SECS` | `10` | DNS peer refresh interval | +| `--full-logs` | `RBFT_FULL_LOGS` | `false` | Emit logs on every consensus cycle | +| `--resend-after-secs` | `RBFT_RESEND_AFTER` | — | Resend messages after N seconds without commits | +| `--disable-express` | `RBFT_DISABLE_EXPRESS` | `false` | Disable express transaction delivery | + +## Testnet environment variables + +| Variable | Default | Description | +|---|---|---| +| `RBFT_NUM_NODES` | `4` | Number of validators | +| `RBFT_GAS_LIMIT` | `600000000` | Block gas limit | +| `RBFT_BLOCK_INTERVAL` | `0.5` | Block time (seconds) | +| `RBFT_EPOCH_LENGTH` | `32` | Blocks per epoch | +| `RBFT_BASE_FEE` | `4761904761905` | Base fee per gas (wei) | +| `RBFT_MAX_ACTIVE_VALIDATORS` | — | Maximum active validators | +| `RBFT_EXIT_AFTER_BLOCK` | — | Stop at this block height | +| `RBFT_ADD_AT_BLOCKS` | — | Add validators at blocks (CSV, e.g., `10,20`) | +| `RBFT_ADD_FOLLOWER_AT` | — | Add followers at blocks (CSV) | +| `RBFT_ADMIN_KEY` | `0x000...0001` | Admin private key | +| `RBFT_DOCKER` | — | Use Docker containers | +| `RBFT_KUBE` | — | Use Kubernetes | +| `RBFT_KUBE_NAMESPACE` | `rbft` | Kubernetes namespace | +| `RBFT_REGISTRY` | — | Container registry for Docker push | +| `RBFT_USE_TRUSTED_PEERS` | — | Write `enodes.txt` for static peers | + +## Log management + +| Variable | Default | Description | +|---|---|---| +| `RBFT_LOG_MAX_SIZE_MB` | `100` | Max log file size before rotation (0 = disabled) | +| `RBFT_LOG_KEEP_ROTATED` | `3` | Number of rotated log files to keep | +| `RBFT_LOGJAM_QUIET` | — | Logjam quiet mode (histogram only) | +| `RBFT_LOGJAM_FOLLOW` | — | Logjam follow mode (like `tail -f`) | +| `RBFT_LOGJAM_MAX_MESSAGE_DELAY` | `1000` | Max delay before reporting (ms) | + +## Genesis parameters + +These are set at chain creation and encoded in `genesis.json`: + +| Parameter | Default | Set via | +|---|---|---| +| Gas limit | `600000000` | `--gas-limit` or `RBFT_GAS_LIMIT` | +| Block interval | `500ms` | `--block-interval` or `RBFT_BLOCK_INTERVAL` | +| Epoch length | `32` | `--epoch-length` or `RBFT_EPOCH_LENGTH` | +| Base fee | `4761904761905` | `--base-fee` or `RBFT_BASE_FEE` | +| Max validators | unlimited | `--max-active-validators` | +| Validator contract | `0x...1001` | `--validator-contract-address` | diff --git a/book/src/operations/monitoring.md b/book/src/operations/monitoring.md new file mode 100644 index 0000000..fea1c92 --- /dev/null +++ b/book/src/operations/monitoring.md @@ -0,0 +1,97 @@ +# Monitoring and Logs + +## Log files + +By default, node logs are written to `~/.rbft/testnet/logs/`. Each node writes +to its own log file (`node0.log`, `node1.log`, etc.). + +Logs are automatically rotated when they exceed 100 MB (configurable via +`RBFT_LOG_MAX_SIZE_MB`), keeping 3 rotated copies. + +## Logjam — log analysis tool + +Logjam merges logs from all nodes in chronological order and produces a message +delivery histogram. + +### Basic usage + +```bash +# View merged logs with delivery histogram +target/release/rbft-utils logjam + +# Quiet mode — histogram and unreceived messages only +target/release/rbft-utils logjam -q + +# Follow mode (like tail -f) +target/release/rbft-utils logjam -f + +# Wire-level tracing +target/release/rbft-utils logjam --trace +``` + +Or via the Makefile: + +```bash +make logjam +make logs # view raw logs +``` + +### Interpreting the log format + +Consensus state is summarized in a compact format: + +``` +[1 h=5 bt=1 rt=0 chain=4/8 r=0 prva=000 i=1.5:P P=234] +``` + +| Field | Meaning | +|---|---| +| `1` | Node index | +| `h=5` | Current height (working on block 5) | +| `bt=1` | Seconds until block timeout | +| `rt=0` | Seconds until round timeout | +| `chain=4/8` | Chain head at height 4, timestamp 8 | +| `r=0` | Current round | +| `prva=000` | Proposal/prepare/commit acceptance flags | +| `i=1.5:P` | Incoming: proposal for height 1, round 5 | +| `P=234` | Prepares received from validators 2, 3, 4 | + +Message type codes: + +| Code | Message | +|---|---| +| `P` | Proposal | +| `p` | Prepare | +| `c` | Commit | +| `r` | Round Change | +| `b` | NewBlock | +| `Q` | BlockRequest | +| `A` | BlockResponse | + +## Validator Inspector + +A terminal UI for monitoring validator state in real time: + +```bash +make validator-inspector +``` + +## Prometheus and Grafana + +The `monitoring/` directory contains Docker Compose configuration for +Prometheus and Grafana: + +```bash +cd monitoring +docker compose up -d +``` + +This sets up: + +- **Prometheus** — scrapes node metrics +- **Grafana** — dashboards for block production, latency, and resource usage + +## Memory monitoring + +The testnet command tracks RSS memory per node and reports it periodically. +Watch for memory growth that could indicate leaks in long-running tests. diff --git a/book/src/operations/troubleshooting.md b/book/src/operations/troubleshooting.md new file mode 100644 index 0000000..acc9a33 --- /dev/null +++ b/book/src/operations/troubleshooting.md @@ -0,0 +1,86 @@ +# Troubleshooting + +## Chain not producing blocks + +**Symptom:** Block height does not increase. + +1. Check that enough validators are running. QBFT requires `2f + 1` validators + online for liveness (e.g., 3 out of 4). +2. Check logs for round change messages (`r` codes), which indicate the + proposer may be offline. +3. Verify peers are connected: + ```bash + curl -s -X POST http://localhost:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' + ``` +4. Ensure all nodes use the same `genesis.json`. + +## Follower not syncing + +**Symptom:** Follower node stays at a low block height. + +1. Verify `--trusted-peers` contains the correct enode URLs. +2. Check that the follower's `--chain` genesis matches the validators. +3. Look for `NewBlock` or `BlockResponse` messages in the follower's log. +4. Ensure network ports are not blocked between follower and validators. + +## Port conflicts + +Each node needs unique ports for P2P, HTTP, and AuthRPC. Default allocations: + +| Node | P2P | HTTP | AuthRPC | +|---|---|---|---| +| 0 | 30303 | 8545 | 10000 | +| 1 | 30304 | 8546 | 10001 | +| 2 | 30305 | 8547 | 10002 | +| 3 | 30306 | 8548 | 10003 | + +If adding nodes, pick ports that don't collide. + +## Permission denied on data directories + +Docker testnet may create root-owned files. Clean up with: + +```bash +make clean +``` + +Or manually: + +```bash +docker run --rm -v ~/.rbft/testnet:/data busybox rm -rf /data/db /data/logs +``` + +## Transaction pool full + +Under heavy load, the transaction pool may reject new transactions. Increase +pool limits: + +```bash +target/release/rbft-node node \ + --txpool.pending-max-count 150000 \ + --txpool.basefee-max-count 150000 \ + --txpool.queued-max-count 150000 \ + ... +``` + +## Validator not participating after registration + +New validators become active at the **next epoch boundary**, not immediately. +Wait for `epochLength` blocks (default: 32) after registration. + +Check current epoch status: + +```bash +target/release/rbft-utils validator status --rpc-url http://localhost:8545 +``` + +## Collecting diagnostics + +For bug reports, collect: + +1. Node logs: `~/.rbft/testnet/logs/` +2. Logjam output: `target/release/rbft-utils logjam -q` +3. Validator status: `target/release/rbft-utils validator status` +4. Block height across nodes: `cast bn --rpc-url http://localhost:854{5,6,7,8}` diff --git a/book/src/running/docker.md b/book/src/running/docker.md new file mode 100644 index 0000000..96c04f1 --- /dev/null +++ b/book/src/running/docker.md @@ -0,0 +1,57 @@ +# Docker Deployment + +RBFT nodes can be built and run as Docker containers. + +## Building the image + +```bash +make docker-build +``` + +This uses a multi-stage Dockerfile with `cargo-chef` for dependency caching. +The resulting image is tagged `rbft-node:testnet`. + +## Running a Docker testnet + +```bash +RBFT_DOCKER=true make testnet_start +``` + +This builds the image and starts validators as Docker containers using +`--network host`. Each container mounts: + +- `/assets` — genesis file and keys (read-only) +- `/data` — node database +- `/logs` — log files + +## Exposed ports + +| Port | Service | +|---|---| +| 8545+ | JSON-RPC (HTTP) | +| 8551+ | Engine API (AuthRPC) | +| 30303+ | P2P (RLPx) | + +## Manual Docker run + +```bash +docker run --rm --name rbft-node-0 \ + --network host \ + -v ~/.rbft/testnet/assets:/assets:ro \ + -v ~/.rbft/testnet/db:/data \ + -v ~/.rbft/testnet/logs:/logs \ + rbft-node:testnet ./rbft-node node \ + --http --http.port 8545 --http.addr 0.0.0.0 \ + --chain /assets/genesis.json \ + --validator-key /assets/validator-key0.txt \ + --p2p-secret-key /assets/p2p-secret-key0.txt \ + --trusted-peers "enode://..." \ + --datadir /data \ + --port 30303 +``` + +## Pushing to a registry + +```bash +RBFT_REGISTRY=your-registry.example.com make docker-tag-registry +``` diff --git a/book/src/running/follower-nodes.md b/book/src/running/follower-nodes.md new file mode 100644 index 0000000..8ffd6ed --- /dev/null +++ b/book/src/running/follower-nodes.md @@ -0,0 +1,70 @@ +# Follower Nodes + +A follower node connects to the validator network, receives blocks, and +maintains a full copy of the chain. It does **not** participate in consensus, +so it can be added or removed at any time without affecting liveness. + +Follower nodes are useful for: + +- Read-only RPC endpoints +- Chain explorers and indexers +- Backup and archival + +## Prerequisites + +- A running RBFT testnet +- The `genesis.json` from the testnet assets directory +- The `nodes.csv` file (for validator enode URLs) + +## Starting a follower + +Extract validator enode URLs from `nodes.csv`: + +```bash +ENODES=$(awk -F',' 'NR>1{printf "%s%s",sep,$5; sep=","}' \ + ~/.rbft/testnet/assets/nodes.csv) +``` + +Start the follower (note: no `--validator-key`): + +```bash +target/release/rbft-node node \ + --chain ~/.rbft/testnet/assets/genesis.json \ + --datadir /tmp/rbft-follower \ + --port 12345 \ + --authrpc.port 8651 \ + --http --http.port 8600 \ + --disable-discovery \ + --trusted-peers "$ENODES" +``` + +## Key flags + +| Flag | Purpose | +|---|---| +| `--chain` | Path to shared `genesis.json` (must match the network) | +| `--datadir` | Fresh directory for the follower's database | +| `--port` | P2P listen port (must not conflict with validators; default starts at 30303) | +| `--authrpc.port` | Engine API port (must not conflict; default starts at 8551) | +| `--disable-discovery` | Prevents discv4/discv5 discovery; peers are set explicitly | +| `--trusted-peers` | Comma-separated enode URLs of validators | + +The absence of `--validator-key` is what makes this a follower. + +## How followers sync + +Followers receive `NewBlock` messages from validators as blocks are committed. +If a follower falls behind (e.g., it was offline), it sends a `BlockRequest` to +peers and receives a `BlockResponse` containing up to 100 blocks at a time, +catching up to the chain tip. + +## Testnet follower test + +The Makefile includes a target that exercises follower behavior automatically: + +```bash +make testnet_follower_test +``` + +This starts 4 validators, adds follower nodes at blocks 5 and 15, runs until +block 30, and exits. diff --git a/book/src/running/kubernetes.md b/book/src/running/kubernetes.md new file mode 100644 index 0000000..5f217c1 --- /dev/null +++ b/book/src/running/kubernetes.md @@ -0,0 +1,69 @@ +# Kubernetes Deployment + +RBFT supports deploying a full testnet to Kubernetes using a StatefulSet. + +## Quick start + +```bash +RBFT_KUBE=true RBFT_NUM_NODES=4 RBFT_MAX_ACTIVE_VALIDATORS=4 \ + make testnet_start +``` + +This builds and pushes the Docker image, creates the namespace (default: `rbft`), +deploys a ConfigMap with genesis and keys, and creates a StatefulSet. + +Press Ctrl+C once pods are running — the chain continues in Kubernetes. + +## Custom namespace + +```bash +RBFT_KUBE_NAMESPACE=rbft-staging \ + RBFT_KUBE=true RBFT_NUM_NODES=4 RBFT_MAX_ACTIVE_VALIDATORS=4 \ + make testnet_start +``` + +## Verifying + +```bash +NS="${RBFT_KUBE_NAMESPACE:-rbft}" +kubectl get pods -n "$NS" -l app=rbft-node -o wide +``` + +Check block production: + +```bash +kubectl logs -n "$NS" rbft-node-0 -f | grep latest_block +``` + +## Stall monitor + +The monitor checks block height every 30 seconds and captures logs if a stall +is detected: + +```bash +kubectl apply -n "$NS" -f scripts/kube/rbft-monitor.yaml +``` + +Optional Slack notifications: + +```bash +kubectl create secret generic rbft-monitor-slack -n "$NS" \ + --from-literal=url='' +``` + +## Load testing (megatx) + +```bash +kubectl apply -n "$NS" -f scripts/kube/rbft-megatx.yaml +``` + +Configure via environment variables in the manifest: + +- `MEGATX_NUM_TXS` — transactions per batch +- `MEGATX_TARGET_TPS` — target transactions per second (0 = unlimited) +- `MEGATX_SLEEP_SECONDS` — pause between batches + +## Retrieving logs from PVC + +See the full Kubernetes runbook at `doc/rbft-kube-runbook.md` for detailed +instructions on log retrieval, parallel instances, and troubleshooting. diff --git a/book/src/running/local-testnet.md b/book/src/running/local-testnet.md new file mode 100644 index 0000000..0f3c935 --- /dev/null +++ b/book/src/running/local-testnet.md @@ -0,0 +1,103 @@ +# Local Testnet + +The quickest way to start an RBFT chain is with `make testnet_start`, which +generates keys, creates a genesis file, and launches a 4-validator network. + +## Quick start + +```bash +make testnet_start +``` + +This will: + +1. Generate validator keys and P2P secret keys +2. Create a `genesis.json` with the QBFTValidatorSet contract +3. Start 4 validator nodes on HTTP ports 8545–8548 +4. Begin producing blocks (default interval: 500 ms) + +Logs are written to `~/.rbft/testnet/logs/` and chain data to `~/.rbft/testnet/db/`. + +## Customizing the testnet + +Use environment variables to change defaults: + +```bash +RBFT_NUM_NODES=7 RBFT_BLOCK_INTERVAL=1.0 make testnet_start +``` + +| Variable | Default | Description | +|---|---|---| +| `RBFT_NUM_NODES` | `4` | Number of validators (minimum 4) | +| `RBFT_BLOCK_INTERVAL` | `0.5` | Block time in seconds | +| `RBFT_GAS_LIMIT` | `600000000` | Block gas limit | +| `RBFT_EPOCH_LENGTH` | `32` | Blocks per epoch | +| `RBFT_BASE_FEE` | `4761904761905` | Base fee per gas (wei) | +| `RBFT_EXIT_AFTER_BLOCK` | — | Stop the testnet at this block height | +| `RBFT_MAX_ACTIVE_VALIDATORS` | — | Cap on active validators | + +## Step-by-step setup + +If you prefer to run each step manually: + +### 1. Generate node keys + +```bash +target/release/rbft-utils node-gen --assets-dir ~/.rbft/testnet/assets +``` + +This creates `nodes.csv` containing validator addresses, private keys, P2P keys, +and enode URLs. + +### 2. Generate genesis + +```bash +target/release/rbft-utils genesis --assets-dir ~/.rbft/testnet/assets +``` + +Compiles the QBFTValidatorSet contract, deploys it into the genesis state, and +writes `genesis.json`. + +### 3. Start the testnet + +```bash +target/release/rbft-utils testnet --init --assets-dir ~/.rbft/testnet/assets +``` + +The `--init` flag clears any previous data directories before starting. + +## Restarting + +To restart with the same keys and genesis (preserving chain state): + +```bash +make testnet_restart +``` + +To start fresh, use `make testnet_start` again — it regenerates everything. + +## Debug mode + +For verbose logging: + +```bash +make testnet_debug +``` + +This sets `RUST_LOG=debug` and builds in debug profile. + +## Verifying block production + +In a separate terminal: + +```bash +# Using cast (Foundry) +cast bn + +# Using curl +curl -s -X POST http://localhost:8545 \ + -H "Content-Type: application/json" \ + -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' +``` + +Block height should increase steadily. diff --git a/book/src/running/validator-management.md b/book/src/running/validator-management.md new file mode 100644 index 0000000..11f176c --- /dev/null +++ b/book/src/running/validator-management.md @@ -0,0 +1,112 @@ +# Validator Management + +Validators can be added or removed dynamically through the QBFTValidatorSet +contract. Changes take effect at the next epoch boundary. + +## Generating validator keys + +```bash +target/release/rbft-utils validator keygen --ip --port 30305 +``` + +This outputs JSON with all required values: + +```json +{ + "validator_address": "0xABCD...", + "validator_private_key": "0x1234...", + "p2p_secret_key": "abcd...", + "enode": "enode://@:30305" +} +``` + +Save the keys to files: + +```bash +echo "0x1234..." > validator-key-new.txt +echo "abcd..." > p2p-secret-key-new.txt +``` + +## Starting a new validator node + +```bash +target/release/rbft-node node \ + --chain ~/.rbft/testnet/assets/genesis.json \ + --datadir /tmp/rbft-new-validator \ + --port 30305 \ + --authrpc.port 8652 \ + --http --http.port 8601 \ + --disable-discovery \ + --p2p-secret-key p2p-secret-key-new.txt \ + --validator-key validator-key-new.txt \ + --trusted-peers "$ENODES" +``` + +The node will sync but not participate in consensus until registered in the +contract. + +## Registering a validator + +The admin key must match the one used at genesis. For a default local testnet, +it is `0x000...0001` (set via `RBFT_ADMIN_KEY`). + +```bash +target/release/rbft-utils validator add \ + --validator-address 0xABCD... \ + --enode "$ENODE" \ + --rpc-url http://localhost:8545 +``` + +To specify the admin key explicitly: + +```bash +target/release/rbft-utils validator add \ + --admin-key \ + --validator-address 0xABCD... \ + --enode "$ENODE" \ + --rpc-url http://localhost:8545 +``` + +The validator becomes active at the next epoch boundary. + +## Removing a validator + +```bash +target/release/rbft-utils validator remove \ + --validator-address 0xABCD... \ + --rpc-url http://localhost:8545 +``` + +## Checking validator status + +```bash +target/release/rbft-utils validator status --rpc-url http://localhost:8545 +``` + +Or via the Makefile: + +```bash +make validator_status +``` + +## Updating chain parameters + +The admin can update chain parameters through the contract: + +```bash +# Set maximum active validators +target/release/rbft-utils validator set-max-active-validators \ + --value 10 --rpc-url http://localhost:8545 + +# Set block interval (milliseconds) +target/release/rbft-utils validator set-block-interval-ms \ + --value 1000 --rpc-url http://localhost:8545 + +# Set epoch length (blocks) +target/release/rbft-utils validator set-epoch-length \ + --value 64 --rpc-url http://localhost:8545 + +# Set base fee (wei) +target/release/rbft-utils validator set-base-fee \ + --value 1000000000 --rpc-url http://localhost:8545 +```