From 193f7ebd4ba7c3b5975fe0dcda4fffe53c9db6a0 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Sun, 30 Jul 2023 22:22:44 +0200 Subject: [PATCH 1/6] [Docs] Document consensus orchestration --- consensus/README.md | 170 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) diff --git a/consensus/README.md b/consensus/README.md index 60e2a2e4d..b634d9df1 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -140,6 +140,176 @@ graph TB J --> note2 ``` +## Synchronization between the consensus processes + +The consensus module currently depends on the `PaceMaker`, `StateSync`, `LeaderElection`, `Networking`. +It has a bootstrapping state where it: + * Initializes connections to the network through a bootstrap node + * Gets the latest block then switches to one of the mutually exclusive modes: `sync` or `consensus` + * Keeps an updated current height (the greatest height seen on the network) + +```mermaid +sequenceDiagram +participant Consensus +participant StateSync +Note over Node: This embeds Networking, FSM +participant Node +participant Network +Node->>Node:Start +Node->>Network:Join +loop + Node->>Network:RequestCurrentHeight + Network-->>Node: + Node->>Node:SwitchMode +end +alt SyncMode + par + Node->>Consensus:Pause + and + Node->>StateSync:Start + Note over Node,StateSync: Details omitted for another diagram + StateSync-->>Node:SyncDone + end +else ConsensusMode + par + Node->>StateSync:Pause + and + Node->>Consensus:Start + Consensus-->>Node:OutOfSync + end +end +``` + +### StateSync Mode + +In this mode the node is behind the latest known-to-network height and will try to catchup by downloading then applying the downloaded blocks (sequentially) to its local state. + +The `download` and `apply` routines may run in parallel, but `apply` may be blocked by the former if the needed block is missing, it will wait until the needed blocks are downloaded. + +#### Download routine + +* The `download` routine is alive as long as `sync` mode is on +* It checks in its persistence for the latest downloaded block and tries to get and add to its persistence all the blocks; up to the network current height +* After downloading, and before inserting the block, basic (stateless) verification is made to the block +* A downloaded and inserted block is a structurally valid block but should by no mean considered valid w.r.t. its validators signatures or transactions within + +```mermaid +sequenceDiagram +participant Persistence +Note over Node: (Download routine) +participant Node +participant Network +par + loop + Note left of Network: Constantly asking for the highest
known block from network + Node->>Network:UpdateNetWorkCurrentHeight + Network-->>Node: + end +and + Node->>Persistence:GetLastDownloadedBlock + Persistence-->>Node: + loop LastDownloadedBlock < NetworkCurrentHeight + Node->>Network:GetBlock + Network-->>Node: + Note left of Network: Currently the node asks all the network
for the block it wants to download + Node->>Persistence:Append + Persistence-->>Node: + Note left of Node: append if new block + Node->>Node:Wait(until new block) + Note right of Node: If latest network block is reached
Wait until it increments to continue + end +end +``` + +### Apply routine + +_Note: We do not detail how individual transactions are applied or how state is derived from them. Just assume that the state (specifically validator set) may be mutated after each block application._ + +* The `apply` routine remains alive as long as `sync` mode is on +* It needs a starting state (genesis, or loaded local state) to start building blocks from +* Each block is validated and applied before moving to the next one +* The block application is a sequential process starting from genesis, applies blocks in order, up till network current height +* Since basic validation is done at download step and assume that the Pocket node trusts its persistence layer, it is safe to skip basic revalidation +* The `sync` mechanism needs to maintain the chain of trust while applying blocks by performing the following: + * Before applying a block `n`, verify that is is signed by a quorum of `n-1`'s validator set (genesis validator set for block `1`) + * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for a new node +* With this chain of trust, a synching node systematically detects crafted blocks + * No malicious or faulty node could inject an alternative block without making at least `2t+1` sign it + * The persistence layer is mainly a cache for the block application, so a node won't restart application from genesis each time it's booted +* When the routine applies `NetworkCurrentHeight`, it signals this so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block + +_Improvement: After applying a block, the server's signature (the one the block has been downloaded from) is replaced by the current node signature. Meaning that from now on, the current node endorse the block validity (when serving the block to other synchers). It could also serve as means to check the integrity of its own database._ + +```mermaid +sequenceDiagram +participant Persistence +participant SyncRoutine +participant BlockApplier +participant State +loop AppliedBlock < NetworkCurrentHeight + SyncRoutine->>Persistence:GetDownloadedBlock + Persistence->>SyncRoutine: + Note right of Persistence: Or genesis if starting from scratch + SyncRoutine->>State:GetValidatorSet + State->>SyncRoutine: + SyncRoutine->>SyncRoutine:Verify(block.qc, ValidatorSet) + SyncRoutine->>BlockApplier:ApplyBlock + BlockApplier->>SyncRoutine: + Note left of BlockApplier: Tells if it's valid block what are the
changes to be made to the state + SyncRoutine->>State:Update + Note left of State: ValidatorSet should be updated here + SyncRoutine->>Persistence:MarkBlockAsApplied + SyncRoutine->>SyncRoutine:Wait(until block downloaded) + Note left of BlockApplier: Wait for the next block to apply
if it's not downloaded yet +end +SyncRoutine->>SyncRoutine:SignalSyncEnd +Note right of SyncRoutine: StateSync has finished
Switch to consensus mode +``` + +## Consensus Mode + +In this mode, the current node is up to date w.r.t. the latest block applied by the network and can start now participating to the consensus process. + +This process is driven by: +* A pace maker, that alerts the consensus flow about key timings in the block production process. + * `MinBlockTime`: The earliest time a leader node could reap the mempool and start proposing a new block + * `RoundTimeout`: How much time a round is allowed to take before calling for another +* A random but deterministic process to elect the leader of each new round + * Given a unique random seed known to all validators and information about the current height and round being validated, any validator is able to know who is the leader of a round without the need to communicate with others + * The leader election strategy aims to give validators a chance of leading the round proportional to their stake + * Fallback to a round robin strategy if probabilistic election doesn't work +* A consensus flow that aims to increment the height (thus block production) of the chain + * See [Block Generation](#block-generation) + +### Example height,round,step increment + +| Height | Round | Step | Comment | +|--------|-------|------|------------------------------------------| +| 1 | 1 | 0 | Initial round, initial block, | +| 1 | 1 | 1 | Enter Prepare step | +| 1 | 1 | 2 | Enter Pre-Commit step | +| 1 | 2 | 0 | Round interrupted, reset step | +| 1 | 2 | 1 | Enter Prepare step again | +| 1 | 2 | 2 | Enter Pre-Commit step again | +| 1 | 2 | 3 | Enter Commit step for the first time | +| 2 | 1 | 0 | Incremented height, reset round and step | + +#### Example invalid increments + +| Height | Round | Step | Comment | +|--------|-------|------|----------------------------------------------------------------| +| 1 | 1 | 2 | Enter Pre-Commit step | +| 1 | 2 | 2 | **Invalid**: If `Round` increments, `Step` has to reset to `0` | + +| Height | Round | Step | Comment | +|--------|-------|------|--------------------------------------------------------------------| +| 1 | 1 | 2 | Enter Pre-Commit step | +| 2 | 0 | 0 | **Invalid**: `Height` only increments when previous Step is at `3` | + +| Height | Round | Step | Comment | +|--------|-------|------|-------------------------------------------------------------------------------| +| 1 | 2 | 3 | Enter Commit step | +| 2 | 2 | 3 | **Invalid**: When `Height` increments, `Round` to `1` and `Step` reset to `0` | ## Implementation From af87895848be01d4ed28b1e382fdfc77c868d14d Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Mon, 31 Jul 2023 05:22:40 +0200 Subject: [PATCH 2/6] small addition on pace maker and global pace --- consensus/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/consensus/README.md b/consensus/README.md index b634d9df1..afcad74ee 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -146,7 +146,7 @@ The consensus module currently depends on the `PaceMaker`, `StateSync`, `LeaderE It has a bootstrapping state where it: * Initializes connections to the network through a bootstrap node * Gets the latest block then switches to one of the mutually exclusive modes: `sync` or `consensus` - * Keeps an updated current height (the greatest height seen on the network) + * Keeps an updated current height (the greatest block height seen on the network) ```mermaid sequenceDiagram @@ -271,9 +271,10 @@ Note right of SyncRoutine: StateSync has finished
Switch to consensus mode In this mode, the current node is up to date w.r.t. the latest block applied by the network and can start now participating to the consensus process. This process is driven by: -* A pace maker, that alerts the consensus flow about key timings in the block production process. +* A pace maker, that alerts the consensus flow about key timings in the block production process * `MinBlockTime`: The earliest time a leader node could reap the mempool and start proposing a new block * `RoundTimeout`: How much time a round is allowed to take before calling for another + * Pacemaker uses a local clock and a node is only aware of the global pace through new round events * A random but deterministic process to elect the leader of each new round * Given a unique random seed known to all validators and information about the current height and round being validated, any validator is able to know who is the leader of a round without the need to communicate with others * The leader election strategy aims to give validators a chance of leading the round proportional to their stake From 30d1115922db81fb43f8eff66adee041a82d8e0d Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Mon, 31 Jul 2023 10:52:07 +0200 Subject: [PATCH 3/6] [Docs] Document PaceMaker behavior --- consensus/README.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/consensus/README.md b/consensus/README.md index afcad74ee..e74f0a005 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -282,6 +282,51 @@ This process is driven by: * A consensus flow that aims to increment the height (thus block production) of the chain * See [Block Generation](#block-generation) +### PaceMaker block proposal delaying + +The pace maker ensures minimum block time production with the aim to have constant block production pace. + +* Adding a delay instead of directly proposing a block makes the the process concurrent. + * It ensures that the block proposal is done only once per new round step +* When the leader gathers enough `NewRound` messages from replicas to propose a block, a first call to build a block is made + * The proposal attempt may happen before `MinBlockTime` which the `PaceMaker` will delay. + * While delayed, more `NewRound` messages may come-in and the node will use the higher QC obtained to propose the block (discards the previous QC). + * If the timer expires before having any block proposal attempt, any call will trigger the block proposal without delay + * If a late message is received after a block is proposed by another call, the late message is discarded + +#### Concurrent requests +```mermaid +sequenceDiagram + participant ReapMempool + participant BlockPrepare + participant Delay + + ReapMempool->>BlockPrepare:RequestPrepare (id=1) + BlockPrepare->>+Delay:Start delay + Note right of Delay: Wait for minBlockTime + ReapMempool->>BlockPrepare:RequestPrepare (id=2) + BlockPrepare--xReapMempool:doNotPrepare (id=1) + Delay->>-BlockPrepare:End delay + BlockPrepare->>ReapMempool:doPrepare (id=2) + ReapMempool->>BlockPrepare: Prepare block +``` + +#### Late requests + +```mermaid +sequenceDiagram + participant ReapMempool + participant BlockPrepare + participant Delay + + BlockPrepare->>+Delay:Start delay + Note right of Delay: Wait for minBlockTime + Delay->>-BlockPrepare:End delay + ReapMempool->>BlockPrepare:doPrepare (id=1) + BlockPrepare->>ReapMempool:RequestPrepare (id=1) + ReapMempool->>BlockPrepare: Prepare block +``` + ### Example height,round,step increment | Height | Round | Step | Comment | From 242362bef773030856f209187c31e1d38835079e Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Tue, 1 Aug 2023 16:48:55 +0200 Subject: [PATCH 4/6] address review changes --- consensus/README.md | 240 +++++++++++++++++++++++++------------------- 1 file changed, 136 insertions(+), 104 deletions(-) diff --git a/consensus/README.md b/consensus/README.md index e74f0a005..21b096d3f 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -12,6 +12,16 @@ This README serves as a guide to the implementation of the [1.0 Pocket's Consens - [Block Validation](#block-validation) - [Consensus Lifecycle](#consensus-lifecycle) - [State Sync](#state-sync) +- [Synchronization between the consensus processes](#synchronization-between-the-consensus-processes) + - [StateSync Mode](#statesync-mode) + - [Download routine](#download-routine) + - [Apply routine](#apply-routine) +- [Consensus Mode](#consensus-mode) + - [PaceMaker block proposal delaying](#pacemaker-block-proposal-delaying) + - [Concurrent requests](#concurrent-requests) + - [Late requests](#late-requests) + - [Example height,round,step increment](#example-heightroundstep-increment) + - [Example invalid increments](#example-invalid-increments) - [Implementation](#implementation) - [Code Organization](#code-organization) - [Testing](#testing) @@ -142,42 +152,46 @@ graph TB ## Synchronization between the consensus processes -The consensus module currently depends on the `PaceMaker`, `StateSync`, `LeaderElection`, `Networking`. +The consensus module currently depends on the `PaceMaker`, `StateSync`, `LeaderElection` and `Networking`. It has a bootstrapping state where it: * Initializes connections to the network through a bootstrap node - * Gets the latest block then switches to one of the mutually exclusive modes: `sync` or `consensus` * Keeps an updated current height (the greatest block height seen on the network) + * Compares network and local current heights, before switching to one of two mutually exclusive modes: `sync` or `consensus` ```mermaid sequenceDiagram -participant Consensus -participant StateSync -Note over Node: This embeds Networking, FSM -participant Node -participant Network -Node->>Node:Start -Node->>Network:Join -loop - Node->>Network:RequestCurrentHeight - Network-->>Node: - Node->>Node:SwitchMode -end -alt SyncMode - par - Node->>Consensus:Pause - and - Node->>StateSync:Start - Note over Node,StateSync: Details omitted for another diagram - StateSync-->>Node:SyncDone + title: Consensus: Internal orchestration + + participant Consensus + participant StateSync + Note over Node: This embeds Networking, FSM + participant Node + participant Network + + Node->>Node:Start + Node->>Network:Join + loop + Node->>Network:RequestCurrentHeight + Network-->>Node: + Node->>Node:SwitchMode end -else ConsensusMode - par - Node->>StateSync:Pause - and - Node->>Consensus:Start - Consensus-->>Node:OutOfSync + + alt SyncMode + par + Node->>Consensus:Pause + and + Node->>StateSync:Start + Note over Node,StateSync: Details omitted for another diagram + StateSync-->>Node:SyncDone + end + else ConsensusMode + par + Node->>StateSync:Pause + and + Node->>Consensus:Start + Consensus-->>Node:OutOfSync + end end -end ``` ### StateSync Mode @@ -195,30 +209,35 @@ The `download` and `apply` routines may run in parallel, but `apply` may be bloc ```mermaid sequenceDiagram -participant Persistence -Note over Node: (Download routine) -participant Node -participant Network -par - loop - Note left of Network: Constantly asking for the highest
known block from network - Node->>Network:UpdateNetWorkCurrentHeight - Network-->>Node: - end -and - Node->>Persistence:GetLastDownloadedBlock - Persistence-->>Node: - loop LastDownloadedBlock < NetworkCurrentHeight - Node->>Network:GetBlock - Network-->>Node: - Note left of Network: Currently the node asks all the network
for the block it wants to download - Node->>Persistence:Append + title: Consensus: Download blocks + participant Persistence + Note over Node: (Download routine) + participant Node + participant Network + + par + loop + Note left of Network: Constantly asking for the highest
known block from network + Node->>Network:UpdateNetworkCurrentHeight + Network-->>Node: + end + and + Node->>Persistence:GetLastDownloadedBlock Persistence-->>Node: - Note left of Node: append if new block - Node->>Node:Wait(until new block) - Note right of Node: If latest network block is reached
Wait until it increments to continue + + loop LastDownloadedBlock < NetworkCurrentHeight + Node->>Network:GetBlock + Network-->>Node: + Note left of Network: Currently the node asks all the network
for the block it wants to download + + Node->>Persistence:Append + Persistence-->>Node: + Note left of Node: append if new block + + Node->>Node:Wait(until new block) + Note right of Node: If latest network block is reached
Wait for a new block to continue + end end -end ``` ### Apply routine @@ -226,44 +245,54 @@ end _Note: We do not detail how individual transactions are applied or how state is derived from them. Just assume that the state (specifically validator set) may be mutated after each block application._ * The `apply` routine remains alive as long as `sync` mode is on -* It needs a starting state (genesis, or loaded local state) to start building blocks from +* It needs a starting state (genesis, or loaded local state) to build blocks from * Each block is validated and applied before moving to the next one -* The block application is a sequential process starting from genesis, applies blocks in order, up till network current height +* The block application is a sequential process starting from genesis, applying blocks up till network current height * Since basic validation is done at download step and assume that the Pocket node trusts its persistence layer, it is safe to skip basic revalidation -* The `sync` mechanism needs to maintain the chain of trust while applying blocks by performing the following: +* The `apply` mechanism needs to maintain a chain of trust while applying blocks by performing the following: * Before applying a block `n`, verify that is is signed by a quorum of `n-1`'s validator set (genesis validator set for block `1`) - * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for a new node -* With this chain of trust, a synching node systematically detects crafted blocks - * No malicious or faulty node could inject an alternative block without making at least `2t+1` sign it - * The persistence layer is mainly a cache for the block application, so a node won't restart application from genesis each time it's booted -* When the routine applies `NetworkCurrentHeight`, it signals this so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block + * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for any new node +* With this chain of trust, a synching node systematically detects invalid blocks + * No malicious or faulty node could inject an alternative block without making at least `2t+1` validators sign it + * The persistence layer is mainly a cache for the block application, so a node won't restart block application from genesis each time it's rebooted +* When the routine applies `NetworkCurrentHeight`, it signals it so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block _Improvement: After applying a block, the server's signature (the one the block has been downloaded from) is replaced by the current node signature. Meaning that from now on, the current node endorse the block validity (when serving the block to other synchers). It could also serve as means to check the integrity of its own database._ ```mermaid sequenceDiagram -participant Persistence -participant SyncRoutine -participant BlockApplier -participant State -loop AppliedBlock < NetworkCurrentHeight - SyncRoutine->>Persistence:GetDownloadedBlock - Persistence->>SyncRoutine: - Note right of Persistence: Or genesis if starting from scratch - SyncRoutine->>State:GetValidatorSet - State->>SyncRoutine: - SyncRoutine->>SyncRoutine:Verify(block.qc, ValidatorSet) - SyncRoutine->>BlockApplier:ApplyBlock - BlockApplier->>SyncRoutine: - Note left of BlockApplier: Tells if it's valid block what are the
changes to be made to the state - SyncRoutine->>State:Update - Note left of State: ValidatorSet should be updated here - SyncRoutine->>Persistence:MarkBlockAsApplied - SyncRoutine->>SyncRoutine:Wait(until block downloaded) - Note left of BlockApplier: Wait for the next block to apply
if it's not downloaded yet -end -SyncRoutine->>SyncRoutine:SignalSyncEnd -Note right of SyncRoutine: StateSync has finished
Switch to consensus mode + title: Consensus: Apply blocks + + participant Persistence + participant SyncRoutine + participant BlockApplier + participant State + + loop AppliedBlock < NetworkCurrentHeight + SyncRoutine->>Persistence:GetDownloadedBlock + Persistence-->>SyncRoutine: + + Note right of Persistence: Or genesis if starting from scratch + SyncRoutine->>State:GetValidatorSet + State-->>SyncRoutine: + + SyncRoutine->>SyncRoutine:Verify(block.qc, ValidatorSet) + + SyncRoutine->>BlockApplier:ApplyBlock + BlockApplier-->>SyncRoutine: + + Note left of BlockApplier: Tells if it's valid block and what are the
changes to be made to the state + SyncRoutine->>State:Update + + Note left of State: ValidatorSet should be updated here + SyncRoutine->>Persistence:MarkBlockAsApplied + + SyncRoutine->>SyncRoutine:Wait(until block downloaded) + Note left of BlockApplier: Wait for the next block to apply
if it's not downloaded yet + + SyncRoutine->>SyncRoutine:If all blocks applied:
SignalSyncEnd + Note right of SyncRoutine: StateSync has finished
Switch to consensus mode + end ``` ## Consensus Mode @@ -273,10 +302,10 @@ In this mode, the current node is up to date w.r.t. the latest block applied by This process is driven by: * A pace maker, that alerts the consensus flow about key timings in the block production process * `MinBlockTime`: The earliest time a leader node could reap the mempool and start proposing a new block - * `RoundTimeout`: How much time a round is allowed to take before calling for another + * `RoundTimeout`: How much time a round is allowed to take before calling for another one * Pacemaker uses a local clock and a node is only aware of the global pace through new round events -* A random but deterministic process to elect the leader of each new round - * Given a unique random seed known to all validators and information about the current height and round being validated, any validator is able to know who is the leader of a round without the need to communicate with others +* A random but deterministic process to elect the leader of each round + * Given a unique random seed known to all validators and information about the current (height, round) being validated, any validator is able to know who is the leader without needing to communicate with others * The leader election strategy aims to give validators a chance of leading the round proportional to their stake * Fallback to a round robin strategy if probabilistic election doesn't work * A consensus flow that aims to increment the height (thus block production) of the chain @@ -284,19 +313,20 @@ This process is driven by: ### PaceMaker block proposal delaying -The pace maker ensures minimum block time production with the aim to have constant block production pace. - +The pace maker ensures minimum block production time with the aim to have a constant production pace. * Adding a delay instead of directly proposing a block makes the the process concurrent. - * It ensures that the block proposal is done only once per new round step -* When the leader gathers enough `NewRound` messages from replicas to propose a block, a first call to build a block is made + * It ensures that the block proposal is done only once after each `NewRound` step +* When the leader gathers enough `NewRound` messages from replicas to propose a block, a first call to propose a block is made * The proposal attempt may happen before `MinBlockTime` which the `PaceMaker` will delay. - * While delayed, more `NewRound` messages may come-in and the node will use the higher QC obtained to propose the block (discards the previous QC). - * If the timer expires before having any block proposal attempt, any call will trigger the block proposal without delay - * If a late message is received after a block is proposed by another call, the late message is discarded + * While delayed, more `NewRound` messages may come-in and the node will use the higher QC obtained by these late messages to propose the block (discards the previous QC). + * If the timer expires before having any block proposal attempt, any call (with enough signatures) will trigger the block proposal without delay + * If a late message is received after a block has already been proposed by another call, the late message is discarded #### Concurrent requests ```mermaid sequenceDiagram + title Pacemaker early concurrent requests + participant ReapMempool participant BlockPrepare participant Delay @@ -315,6 +345,8 @@ sequenceDiagram ```mermaid sequenceDiagram + title: Pacemaker late request + participant ReapMempool participant BlockPrepare participant Delay @@ -331,31 +363,31 @@ sequenceDiagram | Height | Round | Step | Comment | |--------|-------|------|------------------------------------------| -| 1 | 1 | 0 | Initial round, initial block, | -| 1 | 1 | 1 | Enter Prepare step | -| 1 | 1 | 2 | Enter Pre-Commit step | -| 1 | 2 | 0 | Round interrupted, reset step | -| 1 | 2 | 1 | Enter Prepare step again | -| 1 | 2 | 2 | Enter Pre-Commit step again | -| 1 | 2 | 3 | Enter Commit step for the first time | -| 2 | 1 | 0 | Incremented height, reset round and step | +| 1 | 0 | 1 | Initial round, initial block, | +| 1 | 0 | 2 | Enter Prepare step | +| 1 | 0 | 3 | Enter Pre-Commit step | +| 1 | 1 | 1 | Round interrupted, reset step | +| 1 | 1 | 2 | Enter Prepare step again | +| 1 | 1 | 3 | Enter Pre-Commit step again | +| 1 | 1 | 4 | Enter Commit step for the first time | +| 2 | 0 | 1 | Incremented height, reset round and step | #### Example invalid increments | Height | Round | Step | Comment | |--------|-------|------|----------------------------------------------------------------| -| 1 | 1 | 2 | Enter Pre-Commit step | -| 1 | 2 | 2 | **Invalid**: If `Round` increments, `Step` has to reset to `0` | +| 1 | 1 | 3 | Enter Pre-Commit step | +| 1 | 2 | 3 | **Invalid**: If `Round` increments, `Step` has to reset to `1` | | Height | Round | Step | Comment | |--------|-------|------|--------------------------------------------------------------------| -| 1 | 1 | 2 | Enter Pre-Commit step | -| 2 | 0 | 0 | **Invalid**: `Height` only increments when previous Step is at `3` | +| 1 | 1 | 3 | Enter Pre-Commit step | +| 2 | 0 | 4 | **Invalid**: `Height` only increments when previous Step is at `4` | | Height | Round | Step | Comment | |--------|-------|------|-------------------------------------------------------------------------------| -| 1 | 2 | 3 | Enter Commit step | -| 2 | 2 | 3 | **Invalid**: When `Height` increments, `Round` to `1` and `Step` reset to `0` | +| 1 | 2 | 4 | Enter Commit step | +| 2 | 2 | 4 | **Invalid**: When `Height` increments, reset `Round` to `0` and `Step` to `1` | ## Implementation From 164ef29f9dc5c46e472801e9bdd8dcb5256ef7b5 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 4 Aug 2023 11:22:08 +0200 Subject: [PATCH 5/6] address change requests --- consensus/README.md | 62 ++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/consensus/README.md b/consensus/README.md index 21b096d3f..49e94fe77 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -171,8 +171,8 @@ sequenceDiagram Node->>Node:Start Node->>Network:Join loop - Node->>Network:RequestCurrentHeight - Network-->>Node: + Node->>+Network:RequestCurrentHeight + Network-->>-Node: Node->>Node:SwitchMode end @@ -180,16 +180,16 @@ sequenceDiagram par Node->>Consensus:Pause and - Node->>StateSync:Start + Node->>+StateSync:Start Note over Node,StateSync: Details omitted for another diagram - StateSync-->>Node:SyncDone + StateSync-->>-Node:SyncDone end else ConsensusMode par Node->>StateSync:Pause and - Node->>Consensus:Start - Consensus-->>Node:OutOfSync + Node->>+Consensus:Start + Consensus-->>-Node:OutOfSync end end ``` @@ -218,20 +218,20 @@ sequenceDiagram par loop Note left of Network: Constantly asking for the highest
known block from network - Node->>Network:UpdateNetworkCurrentHeight - Network-->>Node: + Node->>+Network:UpdateNetworkCurrentHeight + Network-->>-Node: end and - Node->>Persistence:GetLastDownloadedBlock - Persistence-->>Node: + Node->>+Persistence:GetLastDownloadedBlock + Persistence-->>-Node: loop LastDownloadedBlock < NetworkCurrentHeight - Node->>Network:GetBlock - Network-->>Node: + Node->>+Network:GetBlock + Network-->>-Node: Note left of Network: Currently the node asks all the network
for the block it wants to download - Node->>Persistence:Append - Persistence-->>Node: + Node->>+Persistence:Append + Persistence-->>-Node: Note left of Node: append if new block Node->>Node:Wait(until new block) @@ -247,48 +247,47 @@ _Note: We do not detail how individual transactions are applied or how state is * The `apply` routine remains alive as long as `sync` mode is on * It needs a starting state (genesis, or loaded local state) to build blocks from * Each block is validated and applied before moving to the next one -* The block application is a sequential process starting from genesis, applying blocks up till network current height -* Since basic validation is done at download step and assume that the Pocket node trusts its persistence layer, it is safe to skip basic revalidation +* The block application begins at the genesis block (1) and sequentially processes additional blocks until it arrives at the head of the chain (i.e the most recent block or highest block height) +* Since basic validation is done at the download step and assumes that the Pocket node trusts its persistence layer, it is safe to skip basic re-validation * The `apply` mechanism needs to maintain a chain of trust while applying blocks by performing the following: - * Before applying a block `n`, verify that is is signed by a quorum of `n-1`'s validator set (genesis validator set for block `1`) + * Before applying block at height `h`, verify that it is signed by a quorum from the validator set at height `h-1`; note that the genesis validator set is used for block `1` * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for any new node -* With this chain of trust, a synching node systematically detects invalid blocks +* With this chain of trust form a total of `3t+1` validators, where at least `2t+1` validators are honest and live. A synching node systematically detects invalid blocks * No malicious or faulty node could inject an alternative block without making at least `2t+1` validators sign it * The persistence layer is mainly a cache for the block application, so a node won't restart block application from genesis each time it's rebooted * When the routine applies `NetworkCurrentHeight`, it signals it so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block -_Improvement: After applying a block, the server's signature (the one the block has been downloaded from) is replaced by the current node signature. Meaning that from now on, the current node endorse the block validity (when serving the block to other synchers). It could also serve as means to check the integrity of its own database._ - ```mermaid sequenceDiagram title: Consensus: Apply blocks participant Persistence participant SyncRoutine - participant BlockApplier + Note over ApplyBlockRoutine: This routine is an abstraction
around block application logic + participant ApplyBlockRoutine participant State loop AppliedBlock < NetworkCurrentHeight - SyncRoutine->>Persistence:GetDownloadedBlock - Persistence-->>SyncRoutine: + SyncRoutine->>+Persistence:GetDownloadedBlock + Persistence-->>-SyncRoutine: Note right of Persistence: Or genesis if starting from scratch - SyncRoutine->>State:GetValidatorSet - State-->>SyncRoutine: + SyncRoutine->>+State:GetValidatorSet + State-->>-SyncRoutine: SyncRoutine->>SyncRoutine:Verify(block.qc, ValidatorSet) - SyncRoutine->>BlockApplier:ApplyBlock - BlockApplier-->>SyncRoutine: + SyncRoutine->>+ApplyBlockRoutine:ApplyBlock + ApplyBlockRoutine-->>-SyncRoutine: - Note left of BlockApplier: Tells if it's valid block and what are the
changes to be made to the state + Note left of ApplyBlockRoutine: Tells if it's valid block and what are the
changes to be made to the state SyncRoutine->>State:Update Note left of State: ValidatorSet should be updated here SyncRoutine->>Persistence:MarkBlockAsApplied SyncRoutine->>SyncRoutine:Wait(until block downloaded) - Note left of BlockApplier: Wait for the next block to apply
if it's not downloaded yet + Note left of ApplyBlockRoutine: Wait for the next block to apply
if it's not downloaded yet SyncRoutine->>SyncRoutine:If all blocks applied:
SignalSyncEnd Note right of SyncRoutine: StateSync has finished
Switch to consensus mode @@ -301,9 +300,8 @@ In this mode, the current node is up to date w.r.t. the latest block applied by This process is driven by: * A pace maker, that alerts the consensus flow about key timings in the block production process - * `MinBlockTime`: The earliest time a leader node could reap the mempool and start proposing a new block - * `RoundTimeout`: How much time a round is allowed to take before calling for another one - * Pacemaker uses a local clock and a node is only aware of the global pace through new round events + * `MinBlockTime`: The earliest time a leader node should reap the mempool and start proposing a new block + * `RoundTimeout`: The amount time a replicas wait before starting a new round * A random but deterministic process to elect the leader of each round * Given a unique random seed known to all validators and information about the current (height, round) being validated, any validator is able to know who is the leader without needing to communicate with others * The leader election strategy aims to give validators a chance of leading the round proportional to their stake From 9fa6d51e8c6a0cb1bd6aa936acd790230c370698 Mon Sep 17 00:00:00 2001 From: Redouane Lakrache Date: Fri, 4 Aug 2023 12:33:39 +0200 Subject: [PATCH 6/6] document pacemaker disabling when manual mode is on --- consensus/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/consensus/README.md b/consensus/README.md index 49e94fe77..f7f147e72 100644 --- a/consensus/README.md +++ b/consensus/README.md @@ -254,7 +254,7 @@ _Note: We do not detail how individual transactions are applied or how state is * By applying each block, the validator set is updated (validators joining or leaving), starting from genesis validator set for any new node * With this chain of trust form a total of `3t+1` validators, where at least `2t+1` validators are honest and live. A synching node systematically detects invalid blocks * No malicious or faulty node could inject an alternative block without making at least `2t+1` validators sign it - * The persistence layer is mainly a cache for the block application, so a node won't restart block application from genesis each time it's rebooted + * The persistence layer is used as a resume point for the block application, so a node won't restart block application from genesis each time it's rebooted * When the routine applies `NetworkCurrentHeight`, it signals it so the node could switch to `consensus` mode. Meanwhile, it waits to apply a new downloaded block ```mermaid @@ -314,6 +314,7 @@ This process is driven by: The pace maker ensures minimum block production time with the aim to have a constant production pace. * Adding a delay instead of directly proposing a block makes the the process concurrent. * It ensures that the block proposal is done only once after each `NewRound` step +* Minimum block production time behavior is automatically disabled when the consensus is in debug mode with manual mode enabled (Manual next view triggering) * When the leader gathers enough `NewRound` messages from replicas to propose a block, a first call to propose a block is made * The proposal attempt may happen before `MinBlockTime` which the `PaceMaker` will delay. * While delayed, more `NewRound` messages may come-in and the node will use the higher QC obtained by these late messages to propose the block (discards the previous QC).