geth giving BAD BLOCK and invalid merkle root error - stops working after that

[SyedMuhamadYasir]

System information

Geth version: geth version 1.16.1-stable-12b4131f
CL client & version: Lighthouse v7.1.0-cfb1f73
OS & Version: Ubuntu 22.04.1 LTS
Commit hash : its a stable version,

Expected behaviour

it should just keep syncing, adding new blocks

Actual behaviour

it encounters a bad block and just stops working altogether (see screenshot)

Steps to reproduce the behaviour

nothing particular, it just happens at random to a node..

Logs (in text form)

```
INFO [08-10|20:19:33.818] Forkchoice requested sync to new head    number=35379 hash=bd3f06..61944d finalized=unknown
INFO [08-10|20:19:33.818] Restarting sync cycle                    reason="chain gapped, head: 35360, newHead: 35379"
ERROR[08-10|20:19:33.818] Nil finalized block cannot evict old blobs
ERROR[08-10|20:19:33.864] Beacon backfilling failed                err="syncing canceled (requested)"
WARN [08-10|20:19:33.955] Failed to load old bad blocks            error="pebble: not found"
ERROR[08-10|20:19:33.956]
########## BAD BLOCK #########
Block: 19714 (0x36bd5ebae5d726803f7d7cb7c601658317e114f85bdcc0b25d52c64db9796859)
Error: invalid merkle root (remote: fb1bdf30c99aaca065710ae1b0db7f7b6c3c788d8b69d22803c2be2e0dbc0710 local: b6e260f14a8ce4f2e27990eb0ab9a0059bac31f3ac7176af34772b1e31b6ec2d) dberr: %!w(<nil>)
Platform: geth v1.16.1 go1.24.4 amd64 linux
VCS: 12b4131f-20250702
Chain config: &params.ChainConfig{ChainID:8888, HomesteadBlock:0, DAOForkBlock:<nil>, DAOForkSupport:false, EIP150Block:0, EIP155Block:0, EIP158Block:0, ByzantiumBlock:0, ConstantinopleBlock:0, PetersburgBlock:0, IstanbulBlock:0, MuirGlacierBlock:<nil>, BerlinBlock:0, LondonBlock:0, ArrowGlacierBlock:<nil>, GrayGlacierBlock:<nil>, MergeNetsplitBlock:0, ShanghaiTime:(*uint64)(0xc0007805a8), CancunTime:(*uint64)(0xc0007805b0), PragueTime:(*uint64)(0xc0007805b8), OsakaTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:0, DepositContractAddress:0x4242424242424242424242424242424242424242, EnableVerkleAtGenesis:false, Ethash:(*params.EthashConfig)(nil), Clique:(*params.CliqueConfig)(nil), BlobScheduleConfig:(*params.BlobScheduleConfig)(0xc0004987a0)}
Receipts:
##############################

ERROR[08-10|20:19:33.956] Nil finalized block cannot evict old blobs
ERROR[08-10|20:19:34.000] Beacon backfilling failed                err="retrieved hash chain is invalid: invalid merkle root (remote: fb1bdf30c99aaca065710ae1b0db7f7b6c3c788d8b69d22803c2be2e0dbc0710 local: b6e260f14a8ce4f2e27990eb0ab9a0059bac31f3ac7176af34772b1e31b6ec2d) dberr: %!w(<nil>)"
```

[rjl493456442]

ChainID:8888 it's a private chain, did you guys modify the source code, applying some your protocol changes on top?

[rjl493456442]

Do you run other client implementations in your network, or it's just a pure Geth private network?

[SyedMuhamadYasir]

Hi @rjl493456442

i did not modify any code, we are running a pure Geth private network (which, if i understand correctly, means that all nodes use Geth as the execution layer software?)

the only thing of note is that we have activated Prague (EL) / Electra (CL) fork on the network (this is essential for my work)

To be absolutely sure, I will share the genesis and config.yaml files in the post, so that you can go through them and point out any possible causes

P.S: all nodes use the same version of Geth/Lighthouse

genesis.json

[SyedMuhamadYasir]

Here is the config.yaml that we are using for the network, as embedded code/markdown (Github does not allow .yaml file uploads)

# Extends the mainnet preset
PRESET_BASE: mainnet
CONFIG_NAME: hoodiUZH
# Genesis
# ---------------------------------------------------------------
# `2**14` (= 16,384)
MIN_GENESIS_ACTIVE_VALIDATOR_COUNT: 16
# 2025-Mar-17 12:00:00 PM UTC
MIN_GENESIS_TIME: 1754582392
GENESIS_FORK_VERSION: 0x42424242
GENESIS_DELAY: 60
# Forking
# ---------------------------------------------------------------
# Some forks are disabled for now:
#  - These may be re-assigned to another fork-version later
#  - Temporarily set to max uint64 value: 2**64 - 1

# Altair
ALTAIR_FORK_VERSION: 0x01100000
ALTAIR_FORK_EPOCH: 0
# Merge
BELLATRIX_FORK_VERSION: 0x02200000
BELLATRIX_FORK_EPOCH: 0
TERMINAL_TOTAL_DIFFICULTY: 0
TERMINAL_BLOCK_HASH: 0x5e5a14e4bdb99d0f818436b1180e4868c835004c1b2c747499c33b9ac60e890c
TERMINAL_BLOCK_HASH_ACTIVATION_EPOCH: 0
# Capella
CAPELLA_FORK_VERSION: 0x04400000
CAPELLA_FORK_EPOCH: 0
# DENEB
DENEB_FORK_VERSION: 0x05500000
DENEB_FORK_EPOCH: 0
# Electra
ELECTRA_FORK_VERSION: 0x06600000
ELECTRA_FORK_EPOCH: 0
# Time parameters
# ---------------------------------------------------------------
# 12 seconds
SECONDS_PER_SLOT: 6
# 14 (estimate from Eth1 mainnet)
SECONDS_PER_ETH1_BLOCK: 10
# 2**8 (= 256) epochs ~27 hours
MIN_VALIDATOR_WITHDRAWABILITY_DELAY: 2
# 2**8 (= 256) epochs ~27 hours
SHARD_COMMITTEE_PERIOD: 128
# 2**11 (= 2,048) Eth1 blocks ~8 hours
ETH1_FOLLOW_DISTANCE: 2048
# Validator cycle
# ---------------------------------------------------------------
# 2**2 (= 4)
INACTIVITY_SCORE_BIAS: 4
# 2**4 (= 16)
INACTIVITY_SCORE_RECOVERY_RATE: 16
# 2**4 * 10**9 (= 16,000,000,000) Gwei
EJECTION_BALANCE: 16000000000
# 2**2 (= 4)
MIN_PER_EPOCH_CHURN_LIMIT: 1
# 2**16 (= 65,536)
CHURN_LIMIT_QUOTIENT: 2
# [New in Deneb:EIP7514] 2**3 (= 8)
MAX_PER_EPOCH_ACTIVATION_CHURN_LIMIT: 100
# Fork choice
# ---------------------------------------------------------------
# 40%
PROPOSER_SCORE_BOOST: 40
# 20%
REORG_HEAD_WEIGHT_THRESHOLD: 10
# 160%
REORG_PARENT_WEIGHT_THRESHOLD: 160
# `2` epochs
REORG_MAX_EPOCHS_SINCE_FINALIZATION: 8
# Deposit contract
# ---------------------------------------------------------------
DEPOSIT_CHAIN_ID: 8888
DEPOSIT_NETWORK_ID: 8888
DEPOSIT_CONTRACT_ADDRESS: 0x4242424242424242424242424242424242424242
# Networking
# ---------------------------------------------------------------
# `10 * 2**20` (= 10485760, 10 MiB)
MAX_PAYLOAD_SIZE: 10485760
# `2**10` (= 1024)
MAX_REQUEST_BLOCKS: 256
# `2**8` (= 256)
EPOCHS_PER_SUBNET_SUBSCRIPTION: 64
# `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months)
MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024
# 5s
TTFB_TIMEOUT: 10
# 10s
RESP_TIMEOUT: 20
ATTESTATION_PROPAGATION_SLOT_RANGE: 32
# 500ms
MAXIMUM_GOSSIP_CLOCK_DISPARITY: 1000
MESSAGE_DOMAIN_INVALID_SNAPPY: 0x00000000
MESSAGE_DOMAIN_VALID_SNAPPY: 0x01000000
# 2 subnets per node
SUBNETS_PER_NODE: 1
# 2**8 (= 64)
ATTESTATION_SUBNET_COUNT: 64
ATTESTATION_SUBNET_EXTRA_BITS: 0
# ceillog2(ATTESTATION_SUBNET_COUNT) + ATTESTATION_SUBNET_EXTRA_BITS
ATTESTATION_SUBNET_PREFIX_BITS: 6
# Deneb
# `2**7` (=128)
MAX_REQUEST_BLOCKS_DENEB: 128
# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK
MAX_REQUEST_BLOB_SIDECARS: 768
# `2**12` (= 4096 epochs, ~18 days)
MIN_EPOCHS_FOR_BLOB_SIDECARS_REQUESTS: 4096
# `6`
BLOB_SIDECAR_SUBNET_COUNT: 6
# `uint64(6)`
MAX_BLOBS_PER_BLOCK: 6
# Electra
# 2**7 * 10**9 (= 128,000,000,000)
MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA: 9600000000000
# 2**8 * 10**9 (= 256,000,000,000)
MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT: 9600000000000
# `9`
BLOB_SIDECAR_SUBNET_COUNT_ELECTRA: 9
# `uint64(9)`
MAX_BLOBS_PER_BLOCK_ELECTRA: 9
# MAX_REQUEST_BLOCKS_DENEB * MAX_BLOBS_PER_BLOCK_ELECTRA
MAX_REQUEST_BLOB_SIDECARS_ELECTRA: 1152

[blazejkrzak]

Do you run other client implementations in your network, or it's just a pure Geth private network?

Hey, you have caused a reggression in 2fd77a6a7e07a135e8027980b1411a09339ee479

from

// Flush out any blobs from limbo that are older than the latest finality
	p.limbo.finalize(p.chain.CurrentFinalBlock())

to

if p.chain.Config().IsCancun(p.head.Number, p.head.Time) {
		p.limbo.finalize(p.chain.CurrentFinalBlock())
	}

Your if is meeting criteria when it does not flush previous head during next
Forkchoice requested sync to new head

The chain ID has no impact in here, because you also experience that in devnets from ethereum, so IT MUST BE FIXED.
My recommendation is to revert your changes and think of better solution.
I think that @karalabe knew exactly what he was doing not implementing the cancun check. I hope @rjl493456442 you know as well what impact have you caused.

[blazejkrzak]

Do you run other client implementations in your network, or it's just a pure Geth private network?

Hey, you have caused a reggression in 2fd77a6a7e07a135e8027980b1411a09339ee479

from

// Flush out any blobs from limbo that are older than the latest finality
	p.limbo.finalize(p.chain.CurrentFinalBlock())

to

if p.chain.Config().IsCancun(p.head.Number, p.head.Time) {
		p.limbo.finalize(p.chain.CurrentFinalBlock())
	}

Your if is meeting criteria when it does not flush previous head during next Forkchoice requested sync to new head

The chain ID has no impact in here, because you also experience that in devnets from ethereum, so IT MUST BE FIXED. My recommendation is to revert your changes and think of better solution. I think that @karalabe knew exactly what he was doing not implementing the cancun check. I hope @rjl493456442 you know as well what impact have you caused.

What Is important to add in here is that other clients does not have this problem and did not introduce the if statement go ethereum has, so they propagate different hashes, thus everything falls apart. If you cannot force others use exactly the same logic the error will persist and cause many troubles, thus is another argument for revert

[SyedMuhamadYasir]

Do you run other client implementations in your network, or it's just a pure Geth private network?

Hey, you have caused a reggression in 2fd77a6a7e07a135e8027980b1411a09339ee479
from

// Flush out any blobs from limbo that are older than the latest finality
	p.limbo.finalize(p.chain.CurrentFinalBlock())

to

if p.chain.Config().IsCancun(p.head.Number, p.head.Time) {
		p.limbo.finalize(p.chain.CurrentFinalBlock())
	}

Your if is meeting criteria when it does not flush previous head during next Forkchoice requested sync to new head
The chain ID has no impact in here, because you also experience that in devnets from ethereum, so IT MUST BE FIXED. My recommendation is to revert your changes and think of better solution. I think that @karalabe knew exactly what he was doing not implementing the cancun check. I hope @rjl493456442 you know as well what impact have you caused.

What Is important to add in here is that other clients does not have this problem and did not introduce the if statement go ethereum has, so they propagate different hashes, thus everything falls apart. If you cannot force others use exactly the same logic the error will persist and cause many troubles, thus is another argument for revert

are you saying that these functions are causing the bad block error and the invalid merkle root in my private testnet?

what do you suggest? using Reth to start my private testnet instead of geth?

[blazejkrzak]

Do you run other client implementations in your network, or it's just a pure Geth private network?

Hey, you have caused a reggression in 2fd77a6a7e07a135e8027980b1411a09339ee479
from

// Flush out any blobs from limbo that are older than the latest finality
	p.limbo.finalize(p.chain.CurrentFinalBlock())

to

if p.chain.Config().IsCancun(p.head.Number, p.head.Time) {
		p.limbo.finalize(p.chain.CurrentFinalBlock())
	}

Your if is meeting criteria when it does not flush previous head during next Forkchoice requested sync to new head
The chain ID has no impact in here, because you also experience that in devnets from ethereum, so IT MUST BE FIXED. My recommendation is to revert your changes and think of better solution. I think that @karalabe knew exactly what he was doing not implementing the cancun check. I hope @rjl493456442 you know as well what impact have you caused.

What Is important to add in here is that other clients does not have this problem and did not introduce the if statement go ethereum has, so they propagate different hashes, thus everything falls apart. If you cannot force others use exactly the same logic the error will persist and cause many troubles, thus is another argument for revert

are you saying that these functions are causing the bad block error and the invalid merkle root in my private testnet?

what do you suggest? using Reth to start my private testnet instead of geth?

I suggest @rjl493456442 to fix the issue in the geth core.
Temp solution for you is to run different execution client

[blazejkrzak]

#32463
Try to build from this version, perhaps it helps