Node is lagging responding to json RPC command after a period of time

[ksiezak]

Description
After about a week after start of our node, response time for JSON RPC command is becoming longer and longer, and eventually it is stopping responding altogether. Restart of a node helps.

Steps to Reproduce

1. Start a node
2. Wait about 5-14 days
3. Send some command, for example to get eth_blockNumber

Actual behavior
Response is slow or nothing happens.

Expected behavior
Node responds with a block number, or anything we ask about in a timely manner.

Desktop:

• Operating System: Linux
• Version: 1.30.3
• Installation Method: PPA
• Consensus Client: Prysm 5.2.0

Additional context
ExecStart=/usr/bin/nethermind --Init.StateDbKeyScheme HalfPath --Pruning.Mode None --Pruning.FullPruningTrigger VolumeFreeSpace --Pruning.FullPruningThresholdMb 375810 --Pruning.FullPruningMaxDegreeOfParallelism -1 --Pruning.FullPruningMemoryBudgetMb 16384 --Pruning.CacheMb 2000 --datadir /opt/nethermind/data --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true --JsonRpc.JwtSecretFile=/opt/nethermind/jwt.hex --Metrics.Enabled false --Sync.SnapSync true --Init.MemoryHint 1500000000 --JsonRpc.Host 0.0.0.0 --JsonRpc.EnabledModules [Eth,Subscribe,Trace,TxPool,Web3,Personal,Proof,Net,Parity,Health,Rpc,Debug]

Pruning is only temporary disabled due to other reasons. Previously it was set to Hybrid, and the bug was still present.

Logs
I didn't see any errors or anything other than standard log messages. I just now enabled debug logs, so next time it happens I will try to paste anything useful.

[ksiezak]

I managed to gather log from 30 minutes, in which problem happened, at least 3 times. Logs have 113MB in size. Any idea how to filter them so they can be useful?

[flcl42]

@ksiezak could you send the entire log if possible