Node stuck at block number 0x1a5fc5

[AntonieDavid]

System information

zkEVM Node version: v0.6.5
OS & Version: Ubuntu 22.04.4 LTS
Commit hash : release v0.6.5
Network: Cardona Testnet

Expected behaviour

node is syncing

Actual behaviour

node is not syncing

Steps to reproduce the behaviour

Run docker compose with versions from release v0.6.5. Node will eventually stuck at block number 0x1a5fc5.

Backtrace

2024-04-22T11:21:16.108Z        INFO    synchronizer/synchronizer.go:549        Syncing block 5565421 of 5752646        {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:16.108Z        INFO    synchronizer/synchronizer.go:550        Getting rollup info from block 5565421 to block 5565521 {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:28.450Z        INFO    synchronizer/synchronizer.go:779        [checkReorg function] BlockNumber: 5565420 BlockHash got from L1 provider: 0x2481d642a4f1778030feddf932ba0f4f33ed740d25e35942bc83591de4eb2c73   {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:28.450Z        INFO    synchronizer/synchronizer.go:780        [checkReorg function] latestBlockNumber: 5565420 latestBlockHash already synced: 0x2481d642a4f1778030feddf932ba0f4f33ed740d25e35942bc83591de4eb2c73     {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:28.450Z        ERROR   state/l1infotree.go:73  error add new leaf to the L1InfoTree. Error: mismatched leaf count: 14856, expected: 14857
/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()
/src/state/l1infotree.go:73 github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf()
/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()
/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()
/src/synchronizer/synchronizer.go:671 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()
/src/synchronizer/synchronizer.go:579 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()
/src/synchronizer/synchronizer.go:421 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
/src/cmd/run.go:339 main.runSynchronizer()
        {"pid": 1, "version": "v0.6.5"}
github.com/0xPolygonHermez/zkevm-node/state.(*State).AddL1InfoTreeLeaf
        /src/state/l1infotree.go:73
github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process
        /src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:47
github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process
        /src/synchronizer/actions/processor_manager/processor_manager.go:66
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange
        /src/synchronizer/synchronizer.go:671
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential
        /src/synchronizer/synchronizer.go:579
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
        /src/synchronizer/synchronizer.go:421
main.runSynchronizer
        /src/cmd/run.go:339
2024-04-22T11:21:28.450Z        ERROR   etrog/processor_l1_info_tree_update.go:49       error storing the l1InfoTree(etrog). BlockNumber: 5565443, error: mismatched leaf count: 14856, expected: 14857%!(EXTRA string=
/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
/src/log/log.go:251 github.com/0xPolygonHermez/zkevm-node/log.Errorf()
/src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process()
/src/synchronizer/actions/processor_manager/processor_manager.go:66 github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process()
/src/synchronizer/synchronizer.go:671 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()
/src/synchronizer/synchronizer.go:579 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()
/src/synchronizer/synchronizer.go:421 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
/src/cmd/run.go:339 main.runSynchronizer()
)       {"pid": 1, "version": "v0.6.5"}
github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/etrog.(*ProcessorL1InfoTreeUpdate).Process
        /src/synchronizer/actions/etrog/processor_l1_info_tree_update.go:49
github.com/0xPolygonHermez/zkevm-node/synchronizer/actions/processor_manager.(*L1EventProcessors).Process
        /src/synchronizer/actions/processor_manager/processor_manager.go:66
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange
        /src/synchronizer/synchronizer.go:671
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential
        /src/synchronizer/synchronizer.go:579
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
        /src/synchronizer/synchronizer.go:421
main.runSynchronizer
        /src/cmd/run.go:339
2024-04-22T11:21:28.450Z        ERROR   synchronizer/synchronizer.go:673        error: mismatched leaf count: 14856, expected: 14857
/src/log/log.go:142 github.com/0xPolygonHermez/zkevm-node/log.appendStackTraceMaybeArgs()
/src/log/log.go:217 github.com/0xPolygonHermez/zkevm-node/log.Error()
/src/synchronizer/synchronizer.go:673 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange()
/src/synchronizer/synchronizer.go:579 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential()
/src/synchronizer/synchronizer.go:421 github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync()
/src/cmd/run.go:339 main.runSynchronizer()
        {"pid": 1, "version": "v0.6.5"}
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).ProcessBlockRange
        /src/synchronizer/synchronizer.go:673
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).syncBlocksSequential
        /src/synchronizer/synchronizer.go:579
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
        /src/synchronizer/synchronizer.go:421
main.runSynchronizer
        /src/cmd/run.go:339
2024-04-22T11:21:28.450Z        WARN    synchronizer/synchronizer.go:425        error syncing blocks: mismatched leaf count: 14856, expected: 14857     {"pid": 1, "version": "v0.6.5"}
github.com/0xPolygonHermez/zkevm-node/synchronizer.(*ClientSynchronizer).Sync
        /src/synchronizer/synchronizer.go:425
main.runSynchronizer
        /src/cmd/run.go:339
2024-04-22T11:21:28.451Z        INFO    synchronizer/synchronizer.go:441        L1 state fully synchronized     {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:28.847Z        INFO    synchronizer/synchronizer.go:374        latestSequencedBatchNumber: 76175, latestSyncedBatch: 72614, lastVerifiedBatchNumber: 76173     {"pid": 1, "version": "v0.6.5"}
2024-04-22T11:21:28.847Z        INFO    synchronizer/synchronizer.go:420        Syncing L1 blocks sequentially lastEthBlockSynced=5565420       {"pid": 1, "version": "v0.6.5"}

When submitting logs: please submit them as text and not screenshots.