Circuit Breakers on Subgraph Execution

.changeset/circuit-breaker.md

@@ -0,0 +1,11 @@
+---
+hive-router: patch
+hive-router-plan-executor: patch
+hive-router-config: patch
+---
+
+# Implement Circuit Breaker for Subgraph Requests
+
+This change introduces a circuit breaker mechanism for subgraph requests in the Hive Router. The circuit breaker will monitor the success and failure rates of requests to each subgraph and will prevent future requests if the failure rate exceeds a certain threshold. When the circuit breaker is opened, subsequent requests to that subgraph will fail immediately without attempting to send the request.
+
+This implementation helps improve the resilience and stability of the Hive Router when dealing with unreliable subgraphs.

Cargo.toml

@@ -87,6 +87,7 @@ strum = { version = "0.28.0", features = ["derive"] }
 mockito = "1.7.0"
 futures-util = "0.3.31"
 axum = "0.8.4"
+recloser = "1.3.1"
 notify = "8.2.0"
 
 # Telemetry

docs/README.md

@@ -22,7 +22,7 @@
 |[**subscriptions**](#subscriptions)|`object`|Configuration for subscriptions.<br/>Default: `{"broadcast_capacity":0,"enabled":false}`<br/>||
 |[**supergraph**](#supergraph)|`object`|Configuration for the Federation supergraph source. By default, the router will use a local file-based supergraph source (`./supergraph.graphql`).<br/>||
 |[**telemetry**](#telemetry)|`object`|Default: `{"client_identification":{"name_header":"graphql-client-name","version_header":"graphql-client-version"},"hive":null,"metrics":{"exporters":[],"instrumentation":{"common":{"histogram":{"aggregation":"explicit","bytes":{"buckets":[128,512,1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,3145728,4194304,5242880],"record_min_max":false},"seconds":{"buckets":[0.005,0.01,0.025,0.05,0.075,0.1,0.25,0.5,0.75,1,2.5,5,7.5,10],"record_min_max":false}}},"instruments":{}}},"resource":{"attributes":{}},"tracing":{"collect":{"max_attributes_per_event":16,"max_attributes_per_link":32,"max_attributes_per_span":128,"max_events_per_span":128,"parent_based_sampler":false,"sampling":1},"exporters":[],"instrumentation":{"spans":{"mode":"spec_compliant"}},"propagation":{"b3":false,"baggage":false,"jaeger":false,"trace_context":true}}}`<br/>||
-|[**traffic\_shaping**](#traffic_shaping)|`object`|Configuration for the traffic-shaping of the executor. Use these configurations to control how requests are being executed to subgraphs.<br/>Default: `{"all":{"allow_only_http2":false,"dedupe_enabled":true,"pool_idle_timeout":"50s","request_timeout":"30s"},"max_connections_per_host":100,"router":{"dedupe":{"enabled":false,"headers":"all"},"max_long_lived_clients":128,"request_timeout":"1m"}}`<br/>||
+|[**traffic\_shaping**](#traffic_shaping)|`object`|Configuration for the traffic-shaping of the executor. Use these configurations to control how requests are being executed to subgraphs.<br/>Default: `{"all":{"allow_only_http2":false,"circuit_breaker":null,"dedupe_enabled":true,"pool_idle_timeout":"50s","request_timeout":"30s"},"max_connections_per_host":100,"router":{"dedupe":{"enabled":false,"headers":"all"},"max_long_lived_clients":128,"request_timeout":"1m"}}`<br/>||
 |[**websocket**](#websocket)|`object`|Configuration of router's WebSocket server.<br/>Default: `{"enabled":false,"headers":{"persist":false,"source":"connection"},"path":null}`<br/>||
 
 **Additional Properties:** not allowed  
@@ -203,6 +203,7 @@ telemetry:
 traffic_shaping:
   all:
     allow_only_http2: false
+    circuit_breaker: null
     dedupe_enabled: true
     pool_idle_timeout: 50s
     request_timeout: 30s
@@ -3117,7 +3118,7 @@ Configuration for the traffic-shaping of the executor. Use these configurations
 
 |Name|Type|Description|Required|
 |----|----|-----------|--------|
-|[**all**](#traffic_shapingall)|`object`|The default configuration that will be applied to all subgraphs, unless overridden by a specific subgraph configuration.<br/>Default: `{"allow_only_http2":false,"dedupe_enabled":true,"pool_idle_timeout":"50s","request_timeout":"30s"}`<br/>||
+|[**all**](#traffic_shapingall)|`object`|The default configuration that will be applied to all subgraphs, unless overridden by a specific subgraph configuration.<br/>Default: `{"allow_only_http2":false,"circuit_breaker":null,"dedupe_enabled":true,"pool_idle_timeout":"50s","request_timeout":"30s"}`<br/>||
 |**max\_connections\_per\_host**|`integer`|Limits the concurrent amount of requests/connections per host/subgraph.<br/>Default: `100`<br/>Format: `"uint"`<br/>Minimum: `0`<br/>||
 |[**router**](#traffic_shapingrouter)|`object`|Configuration for the router itself, e.g., for handling incoming requests, or other router-level traffic shaping configurations.<br/>Default: `{"dedupe":{"enabled":false,"headers":"all"},"max_long_lived_clients":128,"request_timeout":"1m"}`<br/>||
 |[**subgraphs**](#traffic_shapingsubgraphs)|`object`|Optional per-subgraph configurations that will override the default configuration for specific subgraphs.<br/>||
@@ -3128,6 +3129,7 @@ Configuration for the traffic-shaping of the executor. Use these configurations
 ```yaml
 all:
   allow_only_http2: false
+  circuit_breaker: null
   dedupe_enabled: true
   pool_idle_timeout: 50s
   request_timeout: 30s
@@ -3152,6 +3154,7 @@ The default configuration that will be applied to all subgraphs, unless overridd
 |Name|Type|Description|Required|
 |----|----|-----------|--------|
 |**allow\_only\_http2**|`boolean`|Forces HTTP/2 for requests to subgraphs.<br/><br/>For plain HTTP, it will use HTTP/2 cleartext (h2c).<br/>For HTTPS, it also requires HTTP/2.<br/>This will make the subgraph requests never fall back to HTTP/1.1,<br/>and will fail if the subgraph doesn't support HTTP/2.<br/>Default: `false`<br/>||
+|[**circuit\_breaker**](#traffic_shapingallcircuit_breaker)|`object`, `null`|Circuit Breaker configuration for all subgraphs.<br/>||
 |**dedupe\_enabled**|`boolean`|Enables/disables request deduplication to subgraphs.<br/><br/>When requests exactly matches the hashing mechanism (e.g., subgraph name, URL, headers, query, variables), and are executed at the same time, they will<br/>be deduplicated by sharing the response of other in-flight requests.<br/>Default: `true`<br/>||
 |**pool\_idle\_timeout**|`string`|Timeout for idle sockets being kept-alive.<br/>Default: `"50s"`<br/>||
 |**request\_timeout**||Optional timeout configuration for requests to subgraphs.<br/><br/>Example with a fixed duration:<br/>```yaml<br/>  timeout:<br/>    duration: 5s<br/>```<br/><br/>Or with a VRL expression that can return a duration based on the operation kind:<br/>```yaml<br/>  timeout:<br/>    expression: \|<br/>     if (.request.operation.type == "mutation") {<br/>       "10s"<br/>     } else {<br/>       "15s"<br/>     }<br/>```<br/>Default: `"30s"`<br/>||
@@ -3162,12 +3165,32 @@ The default configuration that will be applied to all subgraphs, unless overridd
 
 ```yaml
 allow_only_http2: false
+circuit_breaker: null
 dedupe_enabled: true
 pool_idle_timeout: 50s
 request_timeout: 30s
 
 ```
 
+<a name="traffic_shapingallcircuit_breaker"></a>
+#### traffic\_shaping\.all\.circuit\_breaker: object,null
+
+Circuit Breaker configuration for all subgraphs.
+When the circuit breaker is open, requests to the subgraph will be
+short-circuited and an error will be returned to the client.
+The circuit breaker will be triggered based on the error rate of requests to the subgraph, and will attempt to reset after a certain timeout.
+
+
+**Properties**
+
+|Name|Type|Description|Required|
+|----|----|-----------|--------|
+|**enabled**|`boolean`, `null`|Enable or disable the circuit breaker for the subgraph.<br/>Default: false (circuit breaker is disabled)<br/><br/>When unset on a subgraph-level configuration, the value falls back<br/>to the value defined in the global (`all`) circuit breaker<br/>configuration.<br/>||
+|**error\_threshold**|`string`|Percentage after what the circuit breaker should kick in.<br/>Default: 50%<br/>||
+|**reset\_timeout**|`string`|The duration after which the circuit breaker will attempt to retry sending requests to the subgraph.<br/>Default: 30s<br/>||
+|**volume\_threshold**|`integer`, `null`|Count of requests before starting evaluating.<br/>Default: 5<br/>Format: `"uint"`<br/>Minimum: `0`<br/>||
+
+**Additional Properties:** not allowed  
 <a name="traffic_shapingalltls"></a>
 #### traffic\_shaping\.all\.tls: object,null
 
@@ -3280,11 +3303,30 @@ Optional per-subgraph configurations that will override the default configuratio
 |Name|Type|Description|Required|
 |----|----|-----------|--------|
 |**allow\_only\_http2**|`boolean`, `null`|Forces HTTP/2 for requests to subgraphs.<br/><br/>For plain HTTP, it will use HTTP/2 cleartext (h2c).<br/>For HTTPS, it also requires HTTP/2.<br/>This will make the subgraph requests never fall back to HTTP/1.1,<br/>and will fail if the subgraph doesn't support HTTP/2.<br/>||
+|[**circuit\_breaker**](#traffic_shapingsubgraphsadditionalpropertiescircuit_breaker)|`object`, `null`|Circuit Breaker configuration for the subgraph.<br/>||
 |**dedupe\_enabled**|`boolean`, `null`|Enables/disables request deduplication to subgraphs.<br/><br/>When requests exactly matches the hashing mechanism (e.g., subgraph name, URL, headers, query, variables), and are executed at the same time, they will<br/>be deduplicated by sharing the response of other in-flight requests.<br/>||
 |**pool\_idle\_timeout**|`string`, `null`|Timeout for idle sockets being kept-alive.<br/>||
 |**request\_timeout**||Optional timeout configuration for requests to subgraphs.<br/><br/>Example with a fixed duration:<br/>```yaml<br/>  timeout:<br/>    duration: 5s<br/>```<br/><br/>Or with a VRL expression that can return a duration based on the operation kind:<br/>```yaml<br/>  timeout:<br/>    expression: \|<br/>     if (.request.operation.type == "mutation") {<br/>       "10s"<br/>     } else {<br/>       "15s"<br/>     }<br/>```<br/>||
 |[**tls**](#traffic_shapingsubgraphsadditionalpropertiestls)|`object`, `null`|||
 
+**Additional Properties:** not allowed  
+<a name="traffic_shapingsubgraphsadditionalpropertiescircuit_breaker"></a>
+##### traffic\_shaping\.subgraphs\.additionalProperties\.circuit\_breaker: object,null
+
+Circuit Breaker configuration for the subgraph.
+When the circuit breaker is open, requests to the subgraph will be short-circuited and an error will be returned to the client.
+The circuit breaker will be triggered based on the error rate of requests to the subgraph, and will attempt to reset after a certain timeout.
+
+
+**Properties**
+
+|Name|Type|Description|Required|
+|----|----|-----------|--------|
+|**enabled**|`boolean`, `null`|Enable or disable the circuit breaker for the subgraph.<br/>Default: false (circuit breaker is disabled)<br/><br/>When unset on a subgraph-level configuration, the value falls back<br/>to the value defined in the global (`all`) circuit breaker<br/>configuration.<br/>||
+|**error\_threshold**|`string`|Percentage after what the circuit breaker should kick in.<br/>Default: 50%<br/>||
+|**reset\_timeout**|`string`|The duration after which the circuit breaker will attempt to retry sending requests to the subgraph.<br/>Default: 30s<br/>||
+|**volume\_threshold**|`integer`, `null`|Count of requests before starting evaluating.<br/>Default: 5<br/>Format: `"uint"`<br/>Minimum: `0`<br/>||
+
 **Additional Properties:** not allowed  
 <a name="traffic_shapingsubgraphsadditionalpropertiestls"></a>
 ##### traffic\_shaping\.subgraphs\.additionalProperties\.tls: object,null

e2e/configs/circuit_breaker_global.router.yaml

@@ -0,0 +1,13 @@
+# yaml-language-server: $schema=../../router-config.schema.json
+supergraph:
+  source: file
+  path: ../supergraph.graphql
+traffic_shaping:
+  all:
+    circuit_breaker:
+      enabled: true
+      error_threshold: 50%
+      volume_threshold: 5
+      reset_timeout: 30s
+    # Disable deduplication to better test circuit breaker behavior
+    dedupe_enabled: false

e2e/configs/circuit_breaker_mixed.router.yaml

@@ -0,0 +1,18 @@
+# yaml-language-server: $schema=../../router-config.schema.json
+supergraph:
+  source: file
+  path: ../supergraph.graphql
+traffic_shaping:
+  all:
+    circuit_breaker:
+      enabled: true
+      error_threshold: 50%
+      volume_threshold: 10
+      reset_timeout: 30s
+    dedupe_enabled: false
+  subgraphs:
+    accounts:
+      circuit_breaker:
+        enabled: true
+        volume_threshold: 3  # Override only volume_threshold
+        # error_threshold and reset_timeout inherit from global

e2e/configs/circuit_breaker_per_subgraph.router.yaml

@@ -0,0 +1,22 @@
+# yaml-language-server: $schema=../../router-config.schema.json
+supergraph:
+  source: file
+  path: ../supergraph.graphql
+traffic_shaping:
+  all:
+    circuit_breaker:
+      enabled: false  # Disabled globally
+    dedupe_enabled: false
+  subgraphs:
+    accounts:
+      circuit_breaker:
+        enabled: true
+        error_threshold: 60%
+        volume_threshold: 3
+        reset_timeout: 10s
+    products:
+      circuit_breaker:
+        enabled: true
+        error_threshold: 70%
+        volume_threshold: 4
+        reset_timeout: 15s