Revamp NetworkController RPC endpoint events

[mcmire]

Explanation

In a future commit we will introduce changes to network-controller so that it will keep track of the status of each network as requests are made. This commit paves the way for this to happen by redefining the existing RPC endpoint-related events that NetworkController produces.

Currently, when requests are made through the network clients that NetworkController exposes, three events are published:

• NetworkController:rpcEndpointDegraded
  • Published when enough successive retriable errors are encountered while making a request to an RPC endpoint that the maximum number of retries is reached.
• NetworkController:rpcEndpointUnavailable
  • Published when enough successive errors are encountered while making a request to an RPC endpoint that the underlying circuit breaks.
• NetworkController:rpcEndpointRequestRetried
  • Published when a request is retried (mainly used for testing).

It's important to note that in the context of the RPC failover feature, an "RPC endpoint" can actually encompass multiple URLs, so the above events actually fire for any URL.

While these events are useful for reporting metrics on RPC endpoints, in order to effectively be able to update the status of a network, we need events that are less granular and are guaranteed not to fire multiple times in a row. We also need a new event.

Now the list of events looks like this:

• NetworkController:rpcEndpointDegraded
  • The same as before.
• NetworkController:rpcEndpointUnavailable
  • The same as before.
• NetworkController:rpcEndpointRetried
  • Renamed from NetworkController:rpcEndpointRequestRetried.
• NetworkController:rpcEndpointChainDegraded
  • Similar to NetworkController:rpcEndpointDegraded, but won't be published again if the RPC endpoint is already in a degraded state.
• NetworkController:rpcEndpointChainUnavailable
  • Published when all of the circuits underlying all of the URLs for an RPC endpoint have broken (none of the URLs are available). Won't be published again if the RPC endpoint is already in an unavailable state.
• NetworkController:rpcEndpointChainAvailable
  • A new event. Published the first time a successful request is made to one of the URLs for an RPC endpoint, or following a degraded or unavailable status.

Going a bit deeper, in order to make the changes above, it was necessary to rewrite the core logic responsible for diverting traffic to failovers from RpcService to RpcServiceChain, which was a more natural fit, anyway. This also meant that we could simplify RpcService, as well as its tests.

References

Progresses https://consensyssoftware.atlassian.net/browse/WPC-99.

Checklist

• I've updated the test suite for new or updated code as appropriate
• I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
• I've communicated my changes to consumers by updating changelogs for packages I've changed, highlighting breaking changes as necessary
• I've prepared draft pull requests for clients and consumer packages to resolve any breaking changes
  • This will come in a later PR.

---

Note

Introduce chain-level RPC endpoint events and a new RpcServiceChain, rename/update retry and payloads, and refactor failover logic with extensive test updates.

• Events & Payloads (BREAKING):
  • Add NetworkController:rpcEndpointChainAvailable and chain-level events rpcEndpointChainDegraded/rpcEndpointChainUnavailable with non-repeating semantics in src/NetworkController.ts.
  • Rename NetworkController:rpcEndpointRequestRetried to NetworkController:rpcEndpointRetried.
  • Update payloads: add networkClientId, add primaryEndpointUrl to per-endpoint events; chain-level events omit primaryEndpointUrl.
• Failover Architecture:
  • Introduce RpcServiceChain (src/rpc-service/rpc-service-chain.ts) to manage primary/failover endpoints, circuit states, and emit chain/service events.
  • Simplify RpcService (src/rpc-service/rpc-service.ts): remove embedded failover, add onAvailable, resetPolicy, getCircuitState, improved error handling/logging.
  • Wire into client creation: create-network-client.ts builds RpcServiceChain and publishes updated events; create-auto-managed-network-client.ts now passes networkClientId to createNetworkClient.
• Exports & Types:
  • Update exports in src/index.ts; adjust RpcServiceRequestable event listener types and remove { isolated: true } from onBreak data.
• Tests:
  • Add comprehensive tests for new chain/service events and behaviors; update helpers/mocks to support dynamic responses.
• Changelog & Deps:
  • Document breaking changes and new events in CHANGELOG.md.
  • Add cockatiel to devDependencies.

^{Written by Cursor Bugbot for commit 916a0e2. This will update automatically on new commits. Configure here.}

[mcmire]

Prior to these changes, each RpcService object could have an optional failoverService property. This class, RpcServiceChain, would then build a chain (really, a linked list) of services. In order to make a request, RpcServiceChain would call request on the first service in the chain, and RpcService would decide whether it needed to call the next service in the chain, etc.

While this model is easy to understand, I needed access to certain data along the way, and it seemed easier to use a loop rather than a linked list. Anyway, I figured it really should be the responsibility of RpcServiceChain to manage how requests are sent across the chain.

[mcmire]

This should be doing the same thing as the previous code. I just thought that AddToCockatielEventData<Parameters<...>[0] was a bit ugly (and I added some more self-descriptive utility types, so this is one of them).

[mcmire]

We now expose the network client ID in the rpcEndpoint* messenger events, so we need to receive it and pass it down to createNetworkClient.

[mcmire]

These tests got moved to src/create-network-client-tests/rpc-endpoint-events.test.ts as I realized they didn't belong here. It's true that this test file is concerned with the RPC failover feature (which is required to test the rpcEndpoint* events), but all of the tests in tests/network-client directory really exercise the middleware stack that createNetworkClient builds and in so doing loop over all of the RPC methods that our internal provider handles specially. We don't need to go to all that trouble to test the rpcEndpoint* events, we can just use an arbitrary RPC method. (I think eventually I will rename tests/network-client to tests/internal-provider-api or something like that, but that's a PR for another time.)

[mcmire]

The Cockatiel types are a bit awkward to work with (especially since event listeners whose event payloads are empty are typed as Event<void> which is rather strange). These utilities just make them a bit easier to work with.

[mcmire]

In particular, NetworkController:rpcEndpointUnavailable is published only after trying all of the designated URLs for a particular RPC endpoint and the underlying circuit for the last URL breaks, not as each primary's or failover's circuit breaks.

This change is I suppose a bit controversial, and the first version of this was slightly different before I landed on this approach. But I think it makes sense? Basically, if we can reach the network somehow, don't broadcast that it's unavailable until we're absolutely sure. That does mean that the NetworkController:rpcEndpointUnavailable may never be published for Infura networks if the failover does its job, but I think that's precisely the intent.

[mcmire]

Adding no-changelog because there is a minor change to createServicePolicy which is non-user-facing. I've cherry-picked the createServicePolicy to the other PR, so this no longer applies.

[Gudahtt]

LGTM!