etcd panic: assertion failed: Page expected to be: 36312, but self identifies as 0

[Black-max12138]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Etcd fails to be started after the node is powered off and then powered on.
`panic: assertion failed: Page expected to be: 36312, but self identifies as 0

goroutine 293 [running]:
go.etcd.io/bbolt._assert(...)
go.etcd.io/[email protected]/db.go:1387
go.etcd.io/bbolt.(*page).fastCheck(0xfffcb0dd8000, 0x8dd8)
go.etcd.io/[email protected]/page.go:57 +0x184
go.etcd.io/bbolt.(*Tx).page(0x50?, 0xffffada72fe8?)
go.etcd.io/[email protected]/tx.go:534 +0x74
go.etcd.io/bbolt.(*Tx).forEachPageInternal(0x400049a460, {0x40000402d0, 0x1, 0xa}, 0x400029f508)
go.etcd.io/[email protected]/tx.go:546 +0x4c
go.etcd.io/bbolt.(*Tx).forEachPage(...)
go.etcd.io/[email protected]/tx.go:542
go.etcd.io/bbolt.(*Tx).checkBucket(0x400049a460, 0x400049a478, 0x400029f748, 0x400029f688, {0xaaaac8bc65e8, 0xaaaac93ec120}, 0x4000284480)
go.etcd.io/[email protected]/tx_check.go:83 +0xc8
go.etcd.io/bbolt.(*DB).freepages(0x40001e6488)
go.etcd.io/[email protected]/db.go:1205 +0x168
go.etcd.io/bbolt.(*DB).loadFreelist.func1()
go.etcd.io/[email protected]/db.go:417 +0xb4
sync.(*Once).doSlow(0xaaaac83417a0?, 0x40001e6650?)
sync/once.go:74 +0x100
sync.(*Once).Do(...)
sync/once.go:65
go.etcd.io/bbolt.(*DB).loadFreelist(0x40001e6488?)
go.etcd.io/[email protected]/db.go:413 +0x48
go.etcd.io/bbolt.Open({0x40001bb2f0, 0x22}, 0x180, 0x400016eae0)
go.etcd.io/[email protected]/db.go:295 +0x340
go.etcd.io/etcd/server/v3/mvcc/backend.newBackend({{0x40001bb2f0, 0x22}, 0x5f5e100, 0x2710, {0xaaaac8780c41, 0x7}, 0x280000000, 0x4000196190, 0x0, 0x0, ...})
go.etcd.io/etcd/server/v3/mvcc/backend/backend.go:187 +0x15c
go.etcd.io/etcd/server/v3/mvcc/backend.New(...)
go.etcd.io/etcd/server/v3/mvcc/backend/backend.go:163
go.etcd.io/etcd/server/v3/etcdserver.newBackend({{0x4000423d14, 0xc}, {0x0, 0x0}, {0x0, 0x0}, {0x4000427b90, 0x1, 0x1}, {0x4000427a70, ...}, ...}, ...)
go.etcd.io/etcd/server/v3/etcdserver/backend.go:55 +0x2cc
go.etcd.io/etcd/server/v3/etcdserver.openBackend.func1()
go.etcd.io/etcd/server/v3/etcdserver/backend.go:76 +0x50
created by go.etcd.io/etcd/server/v3/etcdserver.openBackend in goroutine 1
go.etcd.io/etcd/server/v3/etcdserver/backend.go:75 +0x120`

What did you expect to happen?

Etcd should start normally.

How can we reproduce it (as minimally and precisely as possible)?

powered off and then powered on.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
# paste output here

$ etcdctl version
# paste output here

3.5.11

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[Black-max12138]

The etcdutl command is used to check whether the file is damaged. However, the etcd-dump-db command cannot be used to read data, which is consistent with the error reported in the etcd log.
etcdutl check broken --data-dir=/opt/etcd/data --check ok
etcd-dump-db list-bucket /opt/etcd/data/ --can not read

[ahrtr]

Unfortunately, the db should have already corrupted. Could you please provide the following info,

• What's the OS and ARCH? Please run uname -a
• What's the OS version?
• What's the filesytem? i.e. ext4, xfs, etc.
• How often do you see this issue? Can it be easily reproduced?

Is it possible to share the db file if it isn't production environment?

[ahrtr]

@Black-max12138 any feedback?