Too many open files - base archive sync

[tmeinlschmidt]

Description
got again into Too many open files, sync stopped

Actual behavior
syncing, had to raise this limit twice

Expected behavior
works without huge file limits

Desktop (please complete the following information):

• linux
• nethermind 1.30.3 (nethermindeth/nethermind:1.30.3-base-fix image)

Additional context
docker - defaults set as

  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 256000,
      "Soft": 128000
    }
  }

command line

./nethermind -c base-mainnet_archive --optimism-sequencerurl https://mainnet-sequencer.base.org \
--jsonrpc-jwtsecretfile /data/jwt.txt --data-dir=/data --jsonrpc-enabled true --jsonrpc-enabledmodules \
 Eth,Subscribe,Debug,Trace,TxPool,Web3,Personal,Proof,Net,Parity,Health,Rpc --jsonrpc-port 18565 \
--jsonrpc-corsorigins * --jsonrpc-maxbatchsize 1024 --jsonrpc-engineport 18561 --network-enableupnp true \ 
--network-maxoutgoingconnectpersec 50 --network-processingthreadcount 32 --sync-tunedbmode HeavyWrite \
--init-memoryhint 6000000000 --metrics-enabled true

Logs

23 Feb 10:49:36 | Syncing previously downloaded blocks from DB (partial offline mode until it finishes)
23 Feb 10:49:37 | Processing part of a long blocks branch 0/2591. Block: 20176456 (0x5883e0...df79c7)
23 Feb 10:49:37 | Non consecutive block commit. This is likely a reorg. Last block commit: 20176456. New block commit: 20176456.
23 Feb 10:49:37 | Encountered exception RocksDbSharp.RocksDbException: IO error: While open a file for random read: /data/nethermind_db/base-mainnet-archive/state/0/2940192.sst: Too many open files
   at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
   at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
   at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145 while processing blocks.
23 Feb 10:49:37 | Processing loop threw an exception. Block: 20179046 (0x570c9c...51253f), Exception: RocksDbSharp.RocksDbException: IO error: While open a file for random read: /data/nethermind_db/base-mainnet-archive/state/0/2940192.sst: Too many open files
   at Nethermind.Db.Rocks.DbOnTheRocks.RocksDbWriteBatch.Dispose() in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 1083
   at Nethermind.Trie.Pruning.TrieStore.FinishBlockCommit(BlockCommitSet set, TrieNode root) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 348
   at Nethermind.State.WorldState.CommitTree(Int64 blockNumber) in /src/Nethermind/Nethermind.State/WorldState.cs:line 181
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Hash256 newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 145
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch& processingBranch, ProcessingOptions options, IBlockTracer tracer, String& error)
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 396
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319