feat: blocks backup / restore #4950

Villaquiranm · 2025-12-08T20:33:02Z

Working on top of #3946
Fixes #1827

Description (thanks @n0izn0iz )

Adds a grpc service called Backup in the tendermint2 node that allows to stream blocks efficiently
It has a single method StreamBlocks that take a start and end height. If end height is 0 it will stream to the latest height. It is disabled by default and require enabling it in the config.toml
Adds a contribs binary named tm2backup that pulls blocks from the backup service and store them in compressed 100-blocks files. It takes a start and end height as well as supporting resuming.
The tar format was chosen to bundle blocks since it's widely supported and efficient. The zstandard format was chosen for compression because it's fast, has a good compression ratio and is widely supported.
Adds a restore subcommand to the gnoland binary that allows to replay blocks from a backup. It takes the options from the start subcommand as well as the backup directory and an optional end height.
It will start at the current node height + 1.

The restore command can only restore at backupEndHeight-1 because I did not figure a way to commit block n without block n+1. I'd gladly take ideas on how to do that.

The backup is fast enough for now IMO (< 20min for test5 on my macbook) but can be optimized because it's not parallelized.
The restore bottleneck seems to be the gnovm currently but I would need to profile to be sure.

How to create a backup

Enable the backup service in your node's config.toml

[backup]
laddr = "localhost:4242"

(Re-)Start your node
Run the tm2backup command

cd contribs/tm2backup
tm2backup -o blocks-backup -remote http://localhost:4242

How to create a node from a backup

Get the genesis file, for example:

wget https://example.com/genesis.json

Run the restore command

gnoland restore --lazy --backup-dir ../contribs/tm2backup/blocks-backup

Start your node

gnoland start

Signed-off-by: Norman <[email protected]>

Gno2D2 · 2025-12-08T20:34:21Z

🛠 PR Checks Summary

🔴 Pending initial approval by a review team member, or review from tech-staff

Manual Checks (for Reviewers):

IGNORE the bot requirements for this PR (force green CI check)

✅ Automated Checks (for Contributors):

🟢 Maintainers must be able to edit this pull request (more info)
🔴 Pending initial approval by a review team member, or review from tech-staff

☑️ Contributor Actions:

Fix any issues flagged by automated checks.
Follow the Contributor Checklist to ensure your PR is ready for review.
- Add new tests, or document why they are unnecessary.
- Provide clear examples/screenshots, if necessary.
- Update documentation, if required.
- Ensure no breaking changes, or include BREAKING CHANGE notes.
- Link related issues/PRs, where applicable.

☑️ Reviewer Actions:

Complete manual checks for the PR, including the guidelines and additional checks if applicable.

📚 Resources:

Debug

Automated Checks

Maintainers must be able to edit this pull request (more info)

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 The pull request was created from a fork (head branch repo: Villaquiranm/gno)

Then

🟢 Requirement satisfied
└── 🟢 Maintainer can modify this pull request

Pending initial approval by a review team member, or review from tech-staff

If

🟢 Condition met
└── 🟢 And
    ├── 🟢 The base branch matches this pattern: ^master$
    └── 🟢 Not (🔴 Pull request author is a member of the team: tech-staff)

Then

🔴 Requirement not satisfied
└── 🔴 If
    ├── 🔴 Condition
    │   └── 🔴 Or
    │       ├── 🔴 At least one of these user(s) reviewed the pull request: [jefft0 leohhhn n0izn0iz notJoon omarsy x1unix] (with state "APPROVED")
    │       ├── 🔴 At least 1 user(s) of the team tech-staff reviewed pull request
    │       └── 🔴 This pull request is a draft
    └── 🔴 Else
        └── 🔴 And
            ├── 🟢 This label is applied to pull request: review/triage-pending
            └── 🔴 On no pull request

Manual Checks

**IGNORE** the bot requirements for this PR (force green CI check)
If
🟢 Condition met
└── 🟢 On every pull request
Can be checked by

Any user with comment edit permission

codecov · 2025-12-08T20:35:38Z

Codecov Report

❌ Patch coverage is 59.68379% with 204 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tm2/pkg/bft/backup/v1/reader.go	0.00%	76 Missing ⚠️
tm2/pkg/bft/backup/v1/writer.go	56.89%	27 Missing and 23 partials ⚠️
gno.land/cmd/gnoland/start.go	38.70%	14 Missing and 5 partials ⚠️
tm2/pkg/bft/backup/v1/util.go	58.97%	9 Missing and 7 partials ⚠️
gno.land/cmd/gnoland/restore.go	83.67%	4 Missing and 4 partials ⚠️
gnovm/pkg/gnolang/string_methods.go	33.33%	6 Missing and 2 partials ⚠️
contribs/tm2backup/main.go	89.55%	5 Missing and 2 partials ⚠️
tm2/pkg/bft/backup/backup_svc.go	85.00%	3 Missing and 3 partials ⚠️
tm2/pkg/bft/blockchain/reactor.go	80.00%	4 Missing and 2 partials ⚠️
.../backup/backuppb/backuppbconnect/backup.connect.go	85.71%	4 Missing ⚠️
... and 1 more

📢 Thoughts on this report? Let us know!

Villaquiranm · 2025-12-10T19:12:21Z

tm2/pkg/bft/backup/backup_svc.go

+	for height := startHeight; height <= endHeight; height++ {
+		block := b.store.LoadBlock(height)
+		if block == nil {
+			return fmt.Errorf("block store returned nil block for height %d", height)
+		}
+
+		data, err := amino.Marshal(block)
+		if err != nil {
+			return err
+		}
+
+		if err := stream.Send(&backuppb.StreamBlocksResponse{Data: data}); err != nil {
+			return err
+		}
+	}


@ajnavarro coment from #3946 (comment)

Some things to comment on this:

Fetching blocks one by one from the K/V storage is extremely slow. We need to use iterators for that.

We are getting byte arrays from the K/V storage, the LoadBlock method is unmarshalling that to a Block struct, after that we marshal again blocks to a byte array. After that, we unmarshal a StreamBlocksResponse into protobuf... That's a lot of unnecessary processing.

After my first tests:
current implementation:
go run main.go -o blocks-backup -remote http://localhost:4242 6.60s user 1.93s system 67% cpu 12.581 total
go run main.go -o blocks-backup -remote http://localhost:4242 7.14s user 2.08s system 68% cpu 13.543 total (over 78.84k blocks)

I will try to improve it with your suggestions, I'll keep you updated

On commit 3382675
We removed the one pair of marshal/unmarshal
and we got:
go run main.go -o blocks-backup -remote http://localhost:4242 3.57s user 2.01s system 40% cpu 13.702 total
Around 2x faster

Villaquiranm · 2025-12-10T19:14:06Z

tm2/pkg/bft/blockchain/reactor.go

+		if !skipVerification {
+			if err := state.Validators.VerifyCommit(
+				chainID, firstID, first.Height, second.LastCommit); err != nil {
+				return fmt.Errorf("invalid commit (%d:%X): %w", first.Height, first.Hash(), err)
+			}


from @ajnavarro comment #3946 (comment)

Can we add a flag to skip commit verification? If we trust the backup, it can improve import speed.

Villaquiranm · 2025-12-10T19:17:34Z

tm2/pkg/bft/blockchain/reactor.go

+			}
+		}
+
+		bcR.store.SaveBlock(first, firstParts, second.LastCommit)


from @ajnavarro comment #3946 (comment):

We have to use insert Batches here. It will be an order of magnitude faster.

n0izn0iz and others added 30 commits March 6, 2025 22:37

feat: backup command

984a889

Signed-off-by: Norman <[email protected]>

feat: use zstd instead of xz

f1a52c8

Signed-off-by: Norman <[email protected]>

chore: move to contribs

ddcb274

Signed-off-by: Norman <[email protected]>

chore: remove worklog

600f085

Signed-off-by: Norman <[email protected]>

chore: revert mischange

47b5677

Signed-off-by: Norman <[email protected]>

chore: improve backupsvc files

ad2c146

Signed-off-by: Norman <[email protected]>

chore: refacto backup service setup

e3ffb38

Signed-off-by: Norman <[email protected]>

fix: changed paths and go mod tidy

6bc40e9

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

af010d7

chore: fail earlier

43cc310

Signed-off-by: Norman <[email protected]>

chore: don't crash if backup config does not existsé

79605c6

Signed-off-by: Norman <[email protected]>

fix: prevent slowloris

a99603f

Signed-off-by: Norman <[email protected]>

chore: format

aae47a3

Signed-off-by: Norman <[email protected]>

fix: populate backup section default

69a2e64

Signed-off-by: Norman <[email protected]>

chore: format

f102c59

Signed-off-by: Norman <[email protected]>

chore: better log for backup server stop

381bdd8

Signed-off-by: Norman <[email protected]>

chore: use constant

82982d9

Signed-off-by: Norman <[email protected]>

feat: restore command

585d637

Signed-off-by: Norman <[email protected]>

chore: lint + refacto

f587f30

Signed-off-by: Norman <[email protected]>

chore: add tm2backup Makefile and move dev tools to devdeps

6524c5b

Signed-off-by: Norman <[email protected]>

Merge branch 'master' into backup-restore

91db151

chore: tidy

4e3a9de

Signed-off-by: Norman <[email protected]>

chore: lint

e55fc15

Signed-off-by: Norman <[email protected]>

chore: remove unused constant

1a0fae4

Signed-off-by: Norman <[email protected]>

chore: Merge remote-tracking branch 'origin/master' into backup-restore

4f9e56c

chore: add backup svc tests

ede2468

Signed-off-by: Norman <[email protected]>

chore: close test server

a6403e0

Signed-off-by: Norman <[email protected]>

chore: close test stream

d516d29

Signed-off-by: Norman <[email protected]>

chore: test backup service init

ad4907a

Signed-off-by: Norman <[email protected]>

chore: remove unneeded field

4002838

Signed-off-by: Norman <[email protected]>

github-project-automation bot added this to 💪 Bounties & Worx and 🧙‍♂️Gno.land development Dec 8, 2025

github-project-automation bot moved this to Triage in 🧙‍♂️Gno.land development Dec 8, 2025

github-actions bot assigned Villaquiranm Dec 8, 2025

Merge branch 'master' into backup-restore

01d184f

github-actions bot added 📦 🤖 gnovm Issues or PRs gnovm related 📦 🌐 tendermint v2 Issues or PRs tm2 related 📦 ⛰️ gno.land Issues or PRs gno.land package related 🐳 devops 🐹 golang Pull requests that update Go code 🛠️ gnodev labels Dec 8, 2025

Villaquiranm changed the title ~~Backup restore~~ feat: blocks backup / restore #3946 Dec 8, 2025

Villaquiranm changed the title ~~feat: blocks backup / restore #3946~~ feat: blocks backup / restore Dec 8, 2025

Villaquiranm added 4 commits December 9, 2025 20:18

fix tests

be98cc0

fmt

79a63b8

add stop application on start cmd

67f3f45

close all apps on cmds

d5c6528

Villaquiranm mentioned this pull request Dec 10, 2025

feat: blocks backup / restore #3946

Open

5 tasks

Villaquiranm added 2 commits December 10, 2025 18:19

Merge branch 'master' into backup-restore

9eb7355

add skip-verification flag

015425f

Villaquiranm marked this pull request as ready for review December 10, 2025 17:55

Gno2D2 added the review/triage-pending PRs opened by external contributors that are waiting for the 1st review label Dec 10, 2025

Villaquiranm commented Dec 10, 2025

View reviewed changes

Villaquiranm added 3 commits December 10, 2025 21:16

remove 1 marshal/unmarshal

3382675

fix tests

0963113

Merge branch 'master' into backup-restore

210c423

feat: blocks backup / restore #4950

Are you sure you want to change the base?

feat: blocks backup / restore #4950

Uh oh!

Conversation

Villaquiranm commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (thanks @n0izn0iz )

How to create a backup

How to create a node from a backup

Uh oh!

Gno2D2 commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛠 PR Checks Summary

Manual Checks (for Reviewers):

✅ Automated Checks (for Contributors):

☑️ Contributor Actions:

☑️ Reviewer Actions:

📚 Resources:

If

Then

If

Then

If

Can be checked by

Uh oh!

codecov bot commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Villaquiranm Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Villaquiranm Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Villaquiranm Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Villaquiranm Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Villaquiranm Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Villaquiranm commented Dec 8, 2025 •

edited

Loading

Gno2D2 commented Dec 8, 2025 •

edited

Loading

codecov bot commented Dec 8, 2025 •

edited

Loading

Villaquiranm Dec 10, 2025 •

edited

Loading

Villaquiranm Dec 10, 2025 •

edited

Loading