evmconnect fails to advance a listener on Polygon mainnet

[nguyer]

I created a new local FireFly stack and pointed it at Polygon mainnet. I attempted to index an existing ERC-20 token on chain and it appears that the listener never advanced and was continually stuck in catchup mode.

I noticed a few of these errors in the evmconnect log

[2022-08-30T18:39:38.569Z] ERROR evmconnect: Failed to query block range fromBlock=16116928 toBlock=16118927 headBlock=32490938: FF22012: Backend RPC request failed eventstream=0182f00c-39d1-5b17-f41e-0e1eac1c8fc6

Full logs attached

ethsigner.log
evmconnect.log
firefly_core.log
tokens.log

[peterbroadhurst]

• EVMConnect logs: We don't have any detail in these errors: FF22012: Backend RPC request failed
  • Maybe the firefly-signer Go module is not upgraded to the latest available in your EVMConnect?
• FireFly Signer: We're only getting return logs (no detail) [2022-08-30T18:37:09.793Z] INFO ffsigner: RPC["000000001"->000000002] <-- [200] OK req=O5wK4Ylr
  • Any chance your FireFly Signer was on info-level logging for this one?

However, from the evidence that is available my best guess is that the JSON/RPC calls to your endpoint are timing out. Because if you take the example of RPC request 000000181 into FireFly Signer, we don't log the reply (not to be confused with the backend request out of FireFly Signer that was allocated 000000181).

See a 24 sec request here from eth_getLogs:

[2022-08-30T18:40:08.882Z] DEBUG evmconnect: RPC[000000181] --> eth_getLogs eventstream=0182f00c-39d1-5b17-f41e-0e1eac1c8fc6
...
[2022-08-30T18:40:32.439Z] ERROR evmconnect: RPC[000000181] <-- ERROR: FF22012: Backend RPC request failed eventstream=0182f00c-39d1-5b17-f41e-0e1eac1c8fc6
...

[nguyer]

Yes, I think FireFly Signer was on info level logging, as I think that's the default that the CLI generates 🙁 Would it be helpful to bump it up to debug logging and test again?

Also, should we (I) change the default to debug for FireFly Signer?

[peterbroadhurst]

I think bumping the default up to debug is a good idea. I'll also submit a PR to tweak the info log to include the JSON RPC method, and the time the call took.

I think you might want to consider some testing with a higher capacity JSON/RPC endpoint as the next step though here - as it seems quite clear it's overloaded to me.

[peterbroadhurst]

I've update INFO logging in firefly-signer pre-req under hyperledger-firefly/signer#23

[peterbroadhurst]

I can add a firefly-signer update to #25 once we have a release there