Stopped responding after persisting from root

[totoCZ]

docker.io/nethermind/nethermind latest-chiseled 48a685a7f778 2 weeks ago 599 MB

Exec=--jsonrpc-enabled true \
--jsonrpc-jwtsecretfile /jwt/jwt.hex \
--jsonrpc-enginehost :: \
--jsonrpc-engineport 8551 \
--jsonrpc-host :: \
--metrics-enabled true \
--metrics-exposehost :: \
--metrics-exposeport 8008 \
--network-discoveryport 7700 \
--network-p2pport 7700 \
--discovery-discoveryversion=all \
--pruning-fullpruningtrigger StateDbSize

Dec 28 23:02:52 systemd[1]: nethermind.service: Killing process 958480 (nethermind) with signal SIGKILL.
Dec 28 23:02:52 systemd[1]: nethermind.service: Killing process 958471 (conmon) with signal SIGKILL.
Dec 28 23:02:52 systemd[1]: nethermind.service: State 'stop-sigterm' timed out. Killing.
Dec 28 23:01:22 systemd[1]: nethermind.service: Stopping timed out. Terminating.
Dec 28 22:59:51 systemd[1]: Stopping Nethermind...
Dec 28 22:39:12 systemd-nethermind[958471]: 28 Dec 22:39:12 | Persisting from root 0xeabbc8d4c9c07df6ef77b7ea3adeefe0b8fffbd56ef6967ba1720703945dac4c in block 21504000
Dec 28 22:39:12 systemd-nethermind[958471]: 28 Dec 22:39:12 | Received New Block:  21504000 (0x0dc656...f35104), Extra Data: Titan (titanbuilder.xyz)
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 | Synced Chain Head to 21503999 (0x31a8b9...564bf7)
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 | Received ForkChoice: 21503999 (0x31a8b9...564bf7), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 |  Block throughput      308.05 MGas/s  |    3,301.4 tps |         32.69 Blk/s | exec code  from cache   1,439 | new      0
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 |  Block mev 0.0173 ETH    9.42 MGas    |      101   txs |  calls    584 ( 41) | sload   1,760 | sstore    606 | create   0
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 | Processed            21503999         |       30.6 ms  |  slot     11,533 ms |<E2> Gas gwei: 5.61 .. 5.61 (6.89) .. 16.84
Dec 28 22:39:01 systemd-nethermind[958471]: 28 Dec 22:39:01 | Received New Block:  21503999 (0x31a8b9...564bf7), Extra Data: Titan (titanbuilder.xyz)
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 | Synced Chain Head to 21503998 (0xc66861...8dccf3)
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 | Received ForkChoice: 21503998 (0xc66861...8dccf3), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 |  Block throughput      374.37 MGas/s  |    3,208.0 tps |         26.08 Blk/s | exec code  from cache   1,574 | new      3
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 |  Block mev 0.0366 ETH   14.35 MGas    |      123   txs |  calls    647 ( 19) | sload   3,693 | sstore    726 | create   0
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 | Processed            21503998         |       38.3 ms  |  slot     13,189 ms |<E2> Gas gwei: 5.64 .. 5.67 (7.22) .. 26.64
Dec 28 22:38:50 systemd-nethermind[958471]: 28 Dec 22:38:50 | Received New Block:  21503998 (0xc66861...8dccf3), Extra Data: beaverbuild.org
Dec 28 22:38:37 systemd-nethermind[958471]: 28 Dec 22:38:37 | Synced Chain Head to 21503997 (0x9c722e...558cc4)
Dec 28 22:38:37 systemd-nethermind[958471]: 28 Dec 22:38:37 | Received ForkChoice: 21503997 (0x9c722e...558cc4), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:38:36 systemd-nethermind[958471]: 28 Dec 22:38:36 |  Block throughput      369.84 MGas/s  |    3,491.4 tps |         26.45 Blk/s | exec code  from cache   1,591 | new      1
Dec 28 22:38:36 systemd-nethermind[958471]: 28 Dec 22:38:36 |  Block mev 0.0261 ETH   13.98 MGas    |      132   txs |  calls    644 ( 24) | sload   2,059 | sstore    703 | create   0
Dec 28 22:38:36 systemd-nethermind[958471]: 28 Dec 22:38:36 | Processed            21503997         |       37.8 ms  |  slot     10,831 ms |<E2> Gas gwei: 5.69 .. 5.78 (6.74) .. 10.71
Dec 28 22:38:36 systemd-nethermind[958471]: 28 Dec 22:38:36 | Received New Block:  21503997 (0x9c722e...558cc4), Extra Data: Titan (titanbuilder.xyz)
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 | Synced Chain Head to 21503996 (0xb944e9...f1a623)
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 | Received ForkChoice: 21503996 (0xb944e9...f1a623), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 |  Block throughput      374.34 MGas/s  |    2,506.9 tps |         15.57 Blk/s | exec code  from cache   2,009 | new      0
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 |  Block mev 0.0331 ETH   24.04 MGas    |      161   txs |  calls    823 ( 28) | sload   4,909 | sstore  1,154 | create   0
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 | Processed            21503996         |       64.2 ms  |  slot     13,580 ms |<E2> Gas gwei: 5.29 .. 5.51 (7.03) .. 34.88
Dec 28 22:38:26 systemd-nethermind[958471]: 28 Dec 22:38:26 | Received New Block:  21503996 (0xb944e9...f1a623), Extra Data: Titan (titanbuilder.xyz)
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 | Synced Chain Head to 21503995 (0xe48274...8b01da)
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 | Received ForkChoice: 21503995 (0xe48274...8b01da), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 |  Block throughput      157.08 MGas/s  |      990.9 tps |          6.61 Blk/s | exec code  from cache   2,848 | new      1
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 |  Block mev 0.0285 ETH   23.78 MGas    |      150   txs |  calls  1,233 ( 45) | sload   3,648 | sstore  1,401 | create  46
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 | Processed            21503995         |      151.4 ms  |  slot     11,479 ms |<E2> Gas gwei: 4.93 .. 4.97 (6.92) .. 45.00
Dec 28 22:38:12 systemd-nethermind[958471]: 28 Dec 22:38:12 | Received New Block:  21503995 (0xe48274...8b01da), Extra Data: beaverbuild.org
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 | Synced Chain Head to 21503994 (0xfe98e8...7606cf)
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 | Received ForkChoice: 21503994 (0xfe98e8...7606cf), Safe: 21503944 (0x1544bc...404cd8), Finalized: 21503912 (0x4c9f96...9d587e)
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 |  Block throughput      362.96 MGas/s  |    3,737.7 tps |         27.48 Blk/s | exec code  from cache   1,313 | new      0
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 |  Block mev 0.0302 ETH   13.21 MGas    |      136   txs |  calls    519 ( 27) | sload   1,821 | sstore    609 | create   0
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 | Processed            21503994         |       36.4 ms  |  slot     12,112 ms |<E2> Gas gwei: 5.01 .. 5.01 (9.91) .. 476.00
Dec 28 22:38:01 systemd-nethermind[958471]: 28 Dec 22:38:01 | Received New Block:  21503994 (0xfe98e8...7606cf), Extra Data: beaverbuild.org

Dec 28 23:10:26 systemd-nethermind[1237773]: 28 Dec 23:10:26 | 8 of 13 checked nodes did not respond to a Ping message - 62 %
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Persisting from root 0xeabbc8d4c9c07df6ef77b7ea3adeefe0b8fffbd56ef6967ba1720703945dac4c in block 21504000
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503999 (0x31a8b9...564bf7)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503998 (0xc66861...8dccf3)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503997 (0x9c722e...558cc4)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503996 (0xb944e9...f1a623)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 |  Block throughput      255.02 MGas/s  |    2,438.1 tps |         15.72 Blk/s | recover     0 | process     5
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 |  Blocks  x16           259.51 MGas    |    2,481   txs |  calls 12,833 (471) | sload  43,963 | sstore 13,523 | create  47
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Processed      21503980... 21503995   |    1,017.6 ms  |  slot      1,032 ms |<E2> Gas gwei: 4.93 .. 4.97 (6.92) .. 45.00
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503995 (0xe48274...8b01da)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503994 (0xfe98e8...7606cf)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503993 (0x60d657...e0ebd8)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503992 (0xb38a55...0de29f)
Dec 28 23:09:56 systemd-nethermind[1237773]: 28 Dec 23:09:56 | Rerunning block after reorg or pruning: 21503991 (0x88da56...1191be)

Workaround:
rm -rf nethermind
systemctl start geth

[kamilchodola]

Hmmm that is interesting - can you provide any hardware details? It looks more like some OOM.

Also seems like there is 20 minutes gap in logs? Can you enable debug logs for Nethermind?