Revamp NetworkController RPC endpoint events

In a future commit we will introduce changes to `network-controller` so
that it will keep track of the status of each network as requests are
made. This commit paves the way for this to happen by redefining the
existing RPC endpoint-related events that NetworkController produces.

Currently, when requests are made through the network clients that
NetworkController exposes, three events are published:

- `NetworkController:rpcEndpointDegraded`
  - Published when enough successive retriable errors are encountered
    while making a request to an RPC endpoint that the maximum number of
    retries is reached.
- `NetworkController:rpcEndpointUnavailable`
  - Published when enough successive errors are encountered while making
    a request to an RPC endpoint that the underlying circuit breaks.
- `NetworkController:rpcEndpointRequestRetried`
  - Published when a request is retried (mainly used for testing).

It's important to note that in the context of the RPC failover feature,
an "RPC endpoint" can actually encompass multiple URLs, so the above
events actually fire for any URL.

While these events are useful for reporting metrics on RPC endpoints, in
order to effectively be able to update the status of a network, we need
events that are less granular and are guaranteed not to fire multiple
times in a row. We also need a new event.

Now the list of events looks like this:

- `NetworkController:rpcEndpointInstanceDegraded`
  - The same as `NetworkController:rpcEndpointDegraded` before.
- `NetworkController:rpcEndpointInstanceUnavailable`
  - The same as `NetworkController:rpcEndpointInstanceDegraded` before.
- `NetworkController:rpcEndpointInstanceRetried`
  - Renamed from `NetworkController:rpcEndpointRequestRetried`.
- `NetworkController:rpcEndpointDegraded`
  - Similar to `NetworkController:rpcEndpointInstanceDegraded`, but
    won't be published again if the RPC endpoint is already in a
    degraded state.
- `NetworkController:rpcEndpointUnavailable`
  - Published when all of the circuits underlying all of the URLs for an
    RPC endpoint have broken (none of the URLs are available). Won't be
    published again if the RPC endpoint is already in an unavailable
    state.
- `NetworkController:rpcEndpointAvailable`
  - A new event. Published the first time a successful request is made
    to one of the URLs for an RPC endpoint, or following a degraded or
    unavailable status.

Author: mcmire

packages/network-controller/CHANGELOG.md

@@ -7,6 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Add `NetworkController:rpcEndpointAvailable` messenger event ([#7166](https://github.com/MetaMask/core/pull/7166))
+  - These are counterparts to the (new) `NetworkController:rpcEndpointUnavailable` and `NetworkController:rpcEndpointDegraded` events, but are published when a request to an RPC endpoint URL is made either initially or following a previously established degraded or unavailable status.
+
 ### Changed
 
 - **BREAKING:** Use `InternalProvider` instead of `SafeEventEmitterProvider` ([#6796](https://github.com/MetaMask/core/pull/6796))
@@ -19,6 +24,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - In practice, this should happen rarely if ever.
 - **BREAKING:** Migrate `NetworkClient` to `JsonRpcEngineV2` ([#7065](https://github.com/MetaMask/core/pull/7065))
   - This ought to be unobservable, but we mark it as breaking out of an abundance of caution.
+- **BREAKING:** Split up and update payload data for `NetworkController:rpcEndpoint{Degraded,Unavailable}` ([#7166](https://github.com/MetaMask/core/pull/7166))
+  - The existing events are now called `NetworkController:rpcEndpointInstance{Degraded,Unavailable}` and retain their present behavior.
+  - `NetworkController:rpcEndpointInstance{Degraded,Unavailable}` do still exist, but they are now designed to represent the entire RPC endpoint and are guaranteed to not be published multiple times in a row. In particular, `NetworkController:rpcEndpointUnavailable` is published only after trying all of the designated URLs for a particular RPC endpoint and the underlying circuit for the last URL breaks, not as each primary's or failover's circuit breaks.
+  - The event payloads have been changed as well: `failoverEndpointUrl` has been renamed to `endpointUrl`, and `endpointUrl` has been renamed to `primaryEndpointUrl`. In addition, `networkClientId` has been added.
+- **BREAKING:** Rename and update payload data for `NetworkController:rpcEndpointRequestRetried` ([#7166](https://github.com/MetaMask/core/pull/7166))
+  - This event is now called `NetworkController:rpcEndpointInstanceRequestRetried`
+  - The event payload has been changed as well: `failoverEndpointUrl` has been renamed to `endpointUrl`, and `endpointUrl` has been renamed to `primaryEndpointUrl`. In addition, `networkClientId` and `attempt` have been added.
+- **BREAKING:** Update `AbstractRpcService`/`RpcServiceRequestable` to remove `{ isolated: true }` from the `onBreak` event data type ([#7166](https://github.com/MetaMask/core/pull/7166))
+  - This represented the error produced when `.isolate` is called on a Cockatiel circuit breaker policy, which we never do.
 - Bump `@metamask/controller-utils` from `^11.14.1` to `^11.15.0` ([#7003](https://github.com/MetaMask/core/pull/7003))
 
 ### Fixed

packages/network-controller/package.json

@@ -78,6 +78,7 @@
     "@types/jest-when": "^2.7.3",
     "@types/lodash": "^4.14.191",
     "@types/node-fetch": "^2.6.12",
+    "cockatiel": "^3.1.2",
     "deep-freeze-strict": "^1.1.1",
     "deepmerge": "^4.2.2",
     "jest": "^27.5.1",

packages/network-controller/src/NetworkController.ts

@@ -443,25 +443,109 @@ export type NetworkControllerNetworkRemovedEvent = {
 };
 
 /**
- * `rpcEndpointUnavailable` is published after an attempt to make a request to
- * an RPC endpoint fails too many times in a row (because of a connection error
- * or an unusable response).
+ * `NetworkController:rpcEndpointUnavailable` is published when the number of
+ * failed consecutive attempts to receive a 2xx response from the primary URL of
+ * an RPC endpoint reaches a maximum, causing further requests to be temporarily
+ * paused, and when subsequent traffic to a failover URL similarly fails.
+ *
+ * In other words, this event will not published if a primary is deemed to be
+ * unavailable but its failover is not.
+ *
+ * Additionally, if this was the last `NetworkController:rpcEndpoint*` event to
+ * be published, the event will not be re-published (for instance, if both a
+ * primary and failover are deemed to be unavailable, or if more than one
+ * failover is deemed to be unavailable).
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the endpoint.
+ * @param payload.endpointUrl - One of the URLs defined for the endpoint which
+ * has been deemed to be unavailable.
+ * @param payload.error - The error from the last request to one of the URLs
+ * defined for the endpoint which determined the unavailability status.
  */
 export type NetworkControllerRpcEndpointUnavailableEvent = {
   type: 'NetworkController:rpcEndpointUnavailable';
   payload: [
     {
       chainId: Hex;
       endpointUrl: string;
-      failoverEndpointUrl?: string;
       error: unknown;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
     },
   ];
 };
 
 /**
- * `rpcEndpointDegraded` is published after a request to an RPC endpoint
- * responds successfully but takes too long.
+ * `NetworkController:rpcEndpointInstanceUnavailable` is published when the
+ * number of failed consecutive attempts to receive a 2xx response from *any* of
+ * the designated URLs of an RPC endpoint reaches a maximum.
+ *
+ * This event will still be published if a primary is deemed to be unavailable,
+ * even its failover is available.
+ *
+ * Additionally, even if this was the last `NetworkController:rpcEndpoint*` event
+ * to be published, the event may be re-published (for instance, if both a
+ * primary and failover are deemed to be unavailable, or if more than one
+ * failover is deemed to be unavailable).
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the endpoint.
+ * @param payload.endpointUrl - One of the URLs defined for the endpoint which
+ * has been deemed to be unavailable.
+ * @param payload.error - The error from the last request to the `endpointUrl`
+ * which determined the unavailability status.
+ */
+export type NetworkControllerRpcEndpointInstanceUnavailableEvent = {
+  type: 'NetworkController:rpcEndpointInstanceUnavailable';
+  payload: [
+    {
+      chainId: Hex;
+      endpointUrl: string;
+      error: unknown;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
+    },
+  ];
+};
+
+/**
+ * `NetworkController:rpcEndpointDegraded` is published in the following two
+ * cases:
+ *
+ * 1. When an attempt to receive a 2xx response from any of the designated URLs
+ * for an RPC endpoint is unsuccessful, and all subsequent automatic retries
+ * lead to the same result.
+ * 2. When a 2xx response is received from any of the endpoint URLs, but the
+ * request takes longer than a set number of seconds to complete.
+ *
+ * Note that this event will be published even if there are local connectivity
+ * issues which prevent requests from being initiated. This is intentional.
+ *
+ * Additionally, if this was the last `NetworkController:rpcEndpoint*` event to
+ * be published, the event will not be re-published (for instance: a failover is
+ * activated and successive attempts to the failover fail, then the primary
+ * comes back online, but it is slow).
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the endpoint.
+ * @param payload.endpointUrl - One of the URLs defined for the endpoint which
+ * has been deemed to be degraded.
+ * @param payload.error - The error from the last request to the `endpointUrl`
+ * which determined the degraded status (or `undefined` if the request was
+ * merely slow).
  */
 export type NetworkControllerRpcEndpointDegradedEvent = {
   type: 'NetworkController:rpcEndpointDegraded';
@@ -470,20 +554,113 @@ export type NetworkControllerRpcEndpointDegradedEvent = {
       chainId: Hex;
       endpointUrl: string;
       error: unknown;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
     },
   ];
 };
 
 /**
- * `rpcEndpointRequestRetried` is published after a request to an RPC endpoint
- * is retried following a connection error or an unusable response.
+ *
+ * `NetworkController:rpcEndpointInstanceDegraded` is published in the following
+ * two cases:
+ *
+ * 1. When an attempt to receive a 2xx response from *any* of the designated
+ * URLs for an RPC endpoint is unsuccessful, and all subsequent automatic
+ * retries lead to the same result.
+ * 2. When a 2xx response is received from any of the endpoint URLs, but the
+ * request takes longer than a set number of seconds to complete.
+ *
+ * Note that this event will be published even if there are local connectivity
+ * issues which prevent requests from being initiated. This is intentional.
+ *
+ * Additionally, if this was the last `NetworkController:rpcEndpoint*` event to
+ * be published, the event may be re-published (for instance: a failover is
+ * activated and successive attempts to the failover fail, then the primary
+ * comes back online, but it is slow).
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the endpoint.
+ * @param payload.endpointUrl - One of the URLs defined for the endpoint which
+ * has been deemed to be degraded.
+ * @param payload.error - The error from the last request to the `endpointUrl`
+ * which determined the degraded status (or `undefined` if the request was
+ * merely slow).
  */
-export type NetworkControllerRpcEndpointRequestRetriedEvent = {
-  type: 'NetworkController:rpcEndpointRequestRetried';
+export type NetworkControllerRpcEndpointInstanceDegradedEvent = {
+  type: 'NetworkController:rpcEndpointInstanceDegraded';
   payload: [
     {
+      chainId: Hex;
       endpointUrl: string;
+      error: unknown;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
+    },
+  ];
+};
+
+/**
+ * `NetworkController:rpcEndpointAvailable` is published in either of the
+ * following two cases:
+ *
+ * 1. The first time that a 2xx request is made to any of the designated URLs of
+ * an RPC endpoint.
+ * 2. When requests to any of the URLs previously failed (placing the endpoint
+ * in a degraded or unavailable status), but are now succeeding again.
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the RPC endpoint.
+ * @param payload.endpointUrl - The specific URL that returned a successful
+ * response.
+ */
+export type NetworkControllerRpcEndpointAvailableEvent = {
+  type: 'NetworkController:rpcEndpointAvailable';
+  payload: [
+    {
+      chainId: Hex;
+      endpointUrl: string;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
+    },
+  ];
+};
+
+/**
+ * `NetworkController:rpcEndpointInstanceRetried` is published before a
+ * request to any of the designated URLs of an RPC endpoint is retried.
+ *
+ * This is mainly useful for tests.
+ *
+ * @param payload - The event payload.
+ * @param payload.chainId - The chain ID of the network where the RPC endpoint
+ * lives.
+ * @param payload.networkClientId - The ID of the network client representing
+ * the RPC endpoint.
+ * @param payload.primaryEndpointUrl - The primary URL of the RPC endpoint.
+ * @param payload.endpointUrl - The URL defined for the endpoint that is being
+ * retried.
+ * @param payload.attempt - The current attempt counter for the endpoint URL
+ * (starting from 0).
+ * @see {@link RpcService} for the list of retriable errors.
+ */
+export type NetworkControllerRpcEndpointInstanceRetriedEvent = {
+  type: 'NetworkController:rpcEndpointInstanceRetried';
+  payload: [
+    {
       attempt: number;
+      chainId: Hex;
+      endpointUrl: string;
+      networkClientId: NetworkClientId;
+      primaryEndpointUrl: string;
     },
   ];
 };
@@ -497,8 +674,11 @@ export type NetworkControllerEvents =
   | NetworkControllerNetworkAddedEvent
   | NetworkControllerNetworkRemovedEvent
   | NetworkControllerRpcEndpointUnavailableEvent
+  | NetworkControllerRpcEndpointInstanceUnavailableEvent
   | NetworkControllerRpcEndpointDegradedEvent
-  | NetworkControllerRpcEndpointRequestRetriedEvent;
+  | NetworkControllerRpcEndpointInstanceDegradedEvent
+  | NetworkControllerRpcEndpointAvailableEvent
+  | NetworkControllerRpcEndpointInstanceRetriedEvent;
 
 /**
  * All events that {@link NetworkController} calls internally.
@@ -2800,6 +2980,7 @@ export class NetworkController extends BaseController<
         autoManagedNetworkClientRegistry[NetworkClientType.Infura][
           addedRpcEndpoint.networkClientId
         ] = createAutoManagedNetworkClient({
+          networkClientId: addedRpcEndpoint.networkClientId,
           networkClientConfiguration: {
             type: NetworkClientType.Infura,
             chainId: networkFields.chainId,
@@ -2818,6 +2999,7 @@ export class NetworkController extends BaseController<
         autoManagedNetworkClientRegistry[NetworkClientType.Custom][
           addedRpcEndpoint.networkClientId
         ] = createAutoManagedNetworkClient({
+          networkClientId: addedRpcEndpoint.networkClientId,
           networkClientConfiguration: {
             type: NetworkClientType.Custom,
             chainId: networkFields.chainId,
@@ -2980,6 +3162,7 @@ export class NetworkController extends BaseController<
           return [
             rpcEndpoint.networkClientId,
             createAutoManagedNetworkClient({
+              networkClientId: rpcEndpoint.networkClientId,
               networkClientConfiguration: {
                 type: NetworkClientType.Infura,
                 network: infuraNetworkName,
@@ -2999,6 +3182,7 @@ export class NetworkController extends BaseController<
         return [
           rpcEndpoint.networkClientId,
           createAutoManagedNetworkClient({
+            networkClientId: rpcEndpoint.networkClientId,
             networkClientConfiguration: {
               type: NetworkClientType.Custom,
               chainId: networkConfiguration.chainId,