Excessive memory consumption (>40G) after upgrading from 1.27.0 to 1.30.3

[Ragnaruk]

Description
After upgrading Nethermind from 1.27.0 to 1.30.3 it started consuming (leaking?) memory at a pretty alarming rate.

I am using Nethermind as an RPC node, and it looks like it is at least somewhat related ­— without any load memory consumption seems stable.

Rate of requests if needed:

eth_getCode: ~2k/min
eth_getTransactionReceipt: ~2k/min
trace_transaction: ~600/min
eth_blockNumber: ~20/min
eth_getBlockByNumber: ~20/min

Screenshots

Desktop (please complete the following information):

• Operating System: Rocky Linux
• Version: 8.10 (Green Obsidian)
• Installation Method: Docker
• Consensus Client: Prysm v5.2.0

[Ragnaruk]

Looks like the problem is with the trace_transaction requests.

Disabling them stabilized memory usage, but didn't reduce it. After a restart, everything returned to normal.

Looks like either the caching is too aggressive or the memory simply isn't released.

[kamilchodola]

Hi @Ragnaruk !

Thanks for the report! We will take a look at that as a priority to fix it asap.

[kamilchodola]

@brbrr Please reproduce it on our infra.

@benaadams Did we had any performance optimizations for tarce calls in last versions?

[brbrr]

@Ragnaruk, could you share some of the trace_transaction requests? that would be greatly appreciated! Also, to confirm - it is happening on mainnet?

[Ragnaruk]

Yes, mainnet.

Here's a quick reproduction:

#!/bin/bash

URL="http://127.0.0.1:8545"
HEADER="Content-Type: application/json"

for i in $(seq 1 100);
do
    block_number=$(curl -s -X POST -H "$HEADER" "$URL" -d '{"jsonrpc":"1.0","id":0,"method":"eth_blockNumber","params":[]}' | jq .result);
    echo "$(date -u) - Block: $block_number";

    transactions=( $(curl -s -X POST -H "$HEADER" "$URL" -d "{\"jsonrpc\":\"1.0\",\"id\":0,\"method\":\"eth_getBlockByNumber\",\"params\":[$block_number]}" | jq .result.transactions | sed -e 's/[][]//g' -e 's/[,]//g' ) );
    echo "$(date -u) - Transactions: ${#transactions[@]}";

    for txhash in "${transactions[@]}";
    do
      curl -s -o /dev/null -X POST -H "$HEADER" "$URL" -d "{\"jsonrpc\":\"1.0\",\"id\":0,\"method\":\"trace_transaction\",\"params\":[$txhash]}"&
      sleep 0.1;
    done
done

[brbrr]

@Ragnaruk thanks! that's very helpful!
I've been running the (a bit modified) stript that you provided, and I definitely see the memory growth. I'll pass it to the developers and hopefully it would be fixed soon

my version of the script:

URL="http://127.0.0.1:8545"
HEADER="Content-Type: application/json"

for i in $(seq 1 500); do
    block_number=$(curl -s -X POST -H "$HEADER" "$URL" -d '{"jsonrpc":"2.0","id":0,"method":"eth_blockNumber","params":[]}' | jq -r .result)
    if [[ -z "$block_number" ]]; then
        echo "$(date -u) - Failed to fetch block number"
        continue
    fi

    transactions=$(curl -s -X POST -H "$HEADER" "$URL" -d "{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"eth_getBlockByNumber\",\"params\":[\"$block_number\" ]}" | jq -r '.result.transactions[]?')
    if [[ -z "$transactions" ]]; then
        echo "$(date -u) - No transactions in block $block_number"
        continue
    fi

    while read -r txhash; do
        trace=$(curl -s -X POST -H "$HEADER" "$URL" -d "{\"jsonrpc\":\"2.0\",\"id\":0,\"method\":\"trace_transaction\",\"params\":[\"$txhash\"]}"&)
        echo "Trace result: $trace"
        sleep 0.1
    done <<< "$transactions"
done