Sepolia Archive node Exception

[stdevMac]

Description
While syncing a

Steps to Reproduce
In order to replicate the behavior, please provide a detailed list of steps:

1. Sync a Sepolia Archive node
2. Got exception, Block: 7520850

Actual behavior

sedge-execution-client  | 20 Jan 09:54:13 | Non consecutive block commit. This is likely a reorg. Last block commit: 7520850. New block commit: 7520850. 
sedge-execution-client  | 20 Jan 09:54:13 | Encountered exception RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145 while processing blocks. 
sedge-execution-client  | 20 Jan 09:54:13 | Processing loop threw an exception. Block: 7520850 (0xaeceac...842177), Exception: RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319 
sedge-execution-client  | 20 Jan 09:54:13 | engine_newPayloadV3 failed: Nethermind.Blockchain.BlockchainException: Block processing threw exception.
sedge-execution-client  |  ---> RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319
sedge-execution-client  |    --- End of inner exception stack trace ---
sedge-execution-client  |    at Nethermind.Merge.Plugin.TimeoutUtils.TimeoutOn[T](Task`1 task, Task timeoutTask) in /src/Nethermind/Nethermind.Merge.Plugin/TimeoutUtils.cs:line 19
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.ValidateBlockAndProcess(Block block, BlockHeader parent, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 379
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.HandleAsync(ExecutionPayload request) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 206
sedge-execution-client  |    at Nethermind.Merge.Plugin.EngineRpcModule.NewPayload(IExecutionPayloadParams executionPayloadParams, Int32 version) in /src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 89 

Expected behavior
To keep syncing

Screenshots
If applicable, please include screenshots to help illustrate the problem.

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: Linux
• Version: release:1.30.2
• Installation Method: Docker
• Consensus Client: consensys/teku:24.10.2

Additional context
Updating the node to 1.30.3 allowed the node to keep syncing.

Logs

sedge-execution-client  | 20 Jan 09:15:27 | Received New Block:  7520850 (0xaeceac...842177), Address: 0x4df6eb2ec570b58cc64f540247a8adfa11f1cf63 
sedge-execution-client  | 20 Jan 09:15:27 | Non consecutive block commit. This is likely a reorg. Last block commit: 7520850. New block commit: 7520850. 
sedge-execution-client  | 20 Jan 09:15:27 | Encountered exception RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145 while processing blocks. 
sedge-execution-client  | 20 Jan 09:15:27 | Processing loop threw an exception. Block: 7520850 (0xaeceac...842177), Exception: RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319 
sedge-execution-client  | 20 Jan 09:15:27 | engine_newPayloadV3 failed: Nethermind.Blockchain.BlockchainException: Block processing threw exception.
sedge-execution-client  |  ---> RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319
sedge-execution-client  |    --- End of inner exception stack trace ---
sedge-execution-client  |    at Nethermind.Merge.Plugin.TimeoutUtils.TimeoutOn[T](Task`1 task, Task timeoutTask) in /src/Nethermind/Nethermind.Merge.Plugin/TimeoutUtils.cs:line 19
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.ValidateBlockAndProcess(Block block, BlockHeader parent, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 379
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.HandleAsync(ExecutionPayload request) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 206
sedge-execution-client  |    at Nethermind.Merge.Plugin.EngineRpcModule.NewPayload(IExecutionPayloadParams executionPayloadParams, Int32 version) in /src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 89 

[LukaszRozmej]

Sounds like RocksDB issue that potentially recovered?

[stdevMac]

sedge-execution-client  | 20 Jan 10:58:48 | Received New Block:  7532181 (0x073223...ef3768), Address: 0xe276bc378a527a8792b353cdca5b5e53263dfb9e 
sedge-execution-client  | 20 Jan 10:58:48 | Non consecutive block commit. This is likely a reorg. Last block commit: 7532181. New block commit: 7532181. 
sedge-execution-client  | 20 Jan 10:58:48 | Encountered exception RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145 while processing blocks. 
sedge-execution-client  | 20 Jan 10:58:48 | Processing loop threw an exception. Block: 7532181 (0x073223...ef3768), Exception: RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319 
sedge-execution-client  | 20 Jan 10:58:48 | engine_newPayloadV3 failed: Nethermind.Blockchain.BlockchainException: Block processing threw exception.
sedge-execution-client  |  ---> RocksDbSharp.RocksDbException: IO error: While pread offset 357986304 len 32397: /nethermind/data/nethermind_db/sepolia_archive/state/0/507519.sst: Input/output error
sedge-execution-client  |    at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
sedge-execution-client  |    at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
sedge-execution-client  |    at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
sedge-execution-client  |    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319
sedge-execution-client  |    --- End of inner exception stack trace ---
sedge-execution-client  |    at Nethermind.Merge.Plugin.TimeoutUtils.TimeoutOn[T](Task`1 task, Task timeoutTask) in /src/Nethermind/Nethermind.Merge.Plugin/TimeoutUtils.cs:line 19
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.ValidateBlockAndProcess(Block block, BlockHeader parent, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 379
sedge-execution-client  |    at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.HandleAsync(ExecutionPayload request) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 206
sedge-execution-client  |    at Nethermind.Merge.Plugin.EngineRpcModule.NewPayload(IExecutionPayloadParams executionPayloadParams, Int32 version) in /src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 89 

[stdevMac]

@LukaszRozmej it started syncing, and after few mins, throw this again

[LukaszRozmej]

Yeah I think this is RocksDB corruption, @asdacap ?