[1/12 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=19379999 min_in_db=19390000

[jdrskr]

Hi,

OS:
Ubuntu 20.04.5 LTS
Erigon:
v2.61.3

I’m running archival node. Recently I upgraded it to the latest version and during the restart I noticed the following warning in the logs:

[WARN] [03-19|09:56:57.969] [1/12 Snapshots] Some blocks are not in snapshots and not in db max_in_snapshots=19379999 min_in_db=19390000

I don't believe this issue is directly related to the upgrade but I tried querying rpc daemon for the blocks in range mentioned above and indeed I can't get any information for these blocks. It's fine for other blocks.

curl -X POST http://localhost:8545/ -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x127B720", true],"id":1}'

{"jsonrpc":"2.0","id":1,"result":null}

Here you can find output from integration print_stages

[INFO] [03-21|13:23:05.346] logging to file system                   log dir=/home/erigon/.local/share/erigon/logs file prefix=integration log level=info json=false
[INFO] [03-21|13:23:05.349] [db] open                                label=chaindata sizeLimit=8TB pageSize=4096
[INFO] [03-21|13:23:10.369] [snapshots:all] Stat                     blocks=19380k indices=19380k alloc=2.9GB sys=2.9GB
[INFO] [03-21|13:23:10.372] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.9GB sys=2.9GB
Note: prune_at doesn't mean 'all data before were deleted' - it just mean stage.Prune function were run to this block. Because 1 stage may prune multiple data types to different prune distance.

 				 stage_at 	 prune_at
Snapshots 			 22095513 	 0
Headers 			 22095514 	 0
BorHeimdall 			 0 		 0
BlockHashes 			 22095514 	 0
Bodies 				 22095514 	 0
Senders 			 22095514 	 0
Execution 			 22095514 	 22095514
Translation 			 0 		 0
HashState 			 22095514 	 0
IntermediateHashes 		 22095514 	 22095514
AccountHistoryIndex 		 22095514 	 0
StorageHistoryIndex 		 22095514 	 0
LogIndex 			 22095514 	 0
CallTraces 			 22095514 	 22095514
TxLookup 			 22095514 	 19386570
Finish 				 22095514 	 0
--
prune distance:

blocks.v2: true, blocks=19379999, segments=19379999, indices=19379999
blocks.bor.v2: segments=0, indices=0

history.v3: false,  idx steps: 0.00, lastBlockInSnap=0, TxNums_Index(0,0)

sequence: EthTx=2783260901, NonCanonicalTx=2145721

in db: first header 19390000, last header 22095514, first body 19387954, last body 22095514

I have seen this issue but given it’s quite old, I wanted to ask if anyone has resolved a similar issue recently and would care to share their experience.

Thanks!

[AskAlexSharov]

integration stage_headers --reset

[jdrskr]

As I understand from previous discussions this process needs to be run while the Erigon node is down, correct?

Can you tell me why it starts to truncate blocks from 20121707, will this help with missing blocks prior to this?

Here are the relevant logs for context:

INFO[03-25|08:31:22.267] logging to file system                   log dir=/home/erigon/.local/share/erigon/logs file prefix=integration log level=info json=false
INFO[03-25|08:31:22.269] [db] open                                label=chaindata sizeLimit=8TB pageSize=8192
INFO[03-25|08:31:25.793] [snapshots:all] Stat                     blocks=20000k indices=20000k alloc=2.9GB sys=3.0GB
INFO[03-25|08:31:25.796] [snapshots:all] Stat                     blocks=0k indices=0k alloc=2.9GB sys=3.0GB
INFO[03-25|08:31:25.913] Disk storage enabled for ethash DAGs     dir=/.local/share/erigon-ethash count=2
INFO[03-25|08:31:46.282] TruncateBlocks                           block=20121707
INFO[03-25|08:32:06.282] TruncateBlocks                           block=20123571
INFO[03-25|08:32:26.290] TruncateBlocks                           block=20125425
INFO[03-25|08:32:46.292] TruncateBlocks                           block=20127195
INFO[03-25|08:33:06.282] TruncateBlocks                           block=20128916
INFO[03-25|08:33:26.286] TruncateBlocks                           block=20130739
INFO[03-25|08:33:46.291] TruncateBlocks                           block=20132569
INFO[03-25|08:34:06.305] TruncateBlocks                           block=20134303
INFO[03-25|08:34:22.569] [mem] memory stats                       Rss=6.3GB Size=0B Pss=6.3GB SharedClean=4.0KB SharedDirty=0B PrivateClean=3.1GB PrivateDirty=3.3GB Referenced=6.3GB Anonymous=3.3GB Swap=0B alloc=2.6GB sys=3.3GB

[jdrskr]

@AskAlexSharov would be great to get your insight on this