System.InvalidOperationException: Operation is not valid due to the current state of the object.

[mikhail-khalizev]

Description
eth_call fails with exception.

Steps to Reproduce
Don't have a good deterministic repro here.

Expected behavior
No error on an eth_call.

Desktop (please complete the following information):

• Operating System: Linux
• Installation Method: Docker
• Version: nethermind/nethermind:1.30.1

Additional context
See also #4002

Logs

xdai-1  | 20 Dec 12:41:04 [JsonRpc.JsonRpcService] Error during method execution, request: Id:0x0, eth_call([{"data":"0x<...>dfb8b"]) System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
xdai-1  |  ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
xdai-1  |    at Nethermind.Consensus.Processing.OverridableTxProcessingEnv.Nethermind.Evm.TransactionProcessing.IOverridableTxProcessorSource.BuildAndOverride(BlockHeader header, Dictionary`2 stateOverride) in /src/Nethermind/Nethermind.Consensus/Processing/OverridableTxProcessingEnv.cs:line 56
xdai-1  |    at Nethermind.Facade.BlockchainBridge.Call(BlockHeader header, Transaction tx, Dictionary`2 stateOverride, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Facade/BlockchainBridge.cs:line 153
xdai-1  |    at Nethermind.JsonRpc.Modules.Eth.ExecutorBase`3.Execute(TRequest call, BlockParameter blockParameter, Dictionary`2 stateOverride) in /src/Nethermind/Nethermind.JsonRpc/Modules/Eth/ExecutorBase.cs:line 31
xdai-1  |    at Nethermind.JsonRpc.Modules.Eth.EthRpcModule.eth_call(TransactionForRpc transactionCall, BlockParameter blockParameter, Dictionary`2 stateOverride) in /src/Nethermind/Nethermind.JsonRpc/Modules/Eth/EthRpcModule.cs:line 331
xdai-1  |    at InvokeStub_IEthRpcModule.eth_call(Object, Span`1)
xdai-1  |    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
xdai-1  |    --- End of inner exception stack trace ---
xdai-1  |    at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
xdai-1  |    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
xdai-1  |    at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ResolvedMethodInfo method, JsonRpcContext context) in /src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 176

[kamilchodola]

Does it fail on every eth_call? Is node somehow loaded with other calls including eth_calls?

[mikhail-khalizev]

I don't know about each eth_call. The node is under load and there are many eth_calls going to it at the same time. And the log consists almost entirely of this error gnosis.logs.txt.zip.

On version 1.29.1 everything works without errors, on 1.30.0, 1.30.1 with this error.

[kamilchodola]

Thanks for the report! Let me check on that

[kamilchodola]

Can't reproduce it for now - maybe you have any more details on how did it happened? I tried hammering my node with thousands of eth calls and just reaching the RPC limit

[mikhail-khalizev]

I use the following docker compose configuration (I don't know if it matters):

  xdai:
    image: nethermind/nethermind:1.30.1
    command:
      --config=gnosis --Init.BaseDbPath /db
      --Network.ExternalIp=${EXTERNAL_IP}
      --JsonRpc.Enabled=true --JsonRpc.Host=0.0.0.0 --JsonRpc.EnginePort=8551 --JsonRpc.EngineHost=0.0.0.0 --JsonRpc.JwtSecretFile=/jwtsecret --JsonRpc.RequestQueueLimit=500000 --JsonRpc.EnabledModules '[Eth, TxPool, Debug, Net, Parity, Personal, Proof, Rpc, Subscribe, Trace, Web3]'
      --Receipt.TxLookupLimit 0

[mikhail-khalizev]

Basically, if you deploy a node so that I can send requests to it, whether directly over the Internet or via VPN (I use ZeroTier), then I can configure it so that my load goes directly to your computer.

[kamilchodola]

Will do and let you know on that