introduce client circuit breaker #115

Tema · 2024-10-31T00:05:47Z

RFC for tikv/pd#8678

Rendered view: https://github.com/Tema/rfcs/blob/0115-circuit-breaker/text/0115-circuit-breaker.md

Tema · 2024-10-31T00:07:19Z

@rleungx @niubell PTAL

Tema · 2024-10-31T00:12:51Z

cc: @mittalrishabh @HaoW30

rleungx · 2024-10-31T02:56:11Z

text/0115-circuit-breaker.md

+# Title
+
+- RFC PR: https://github.com/tikv/rfcs/pull/115
+- Tracking Issue: https://github.com/tikv/repo/issues/0000


We can use this issue: tikv/pd#8678

Signed-off-by: artem_danilov <[email protected]>

rleungx · 2024-11-05T02:24:32Z

@nolouch @okJiang Please also take a look.

rleungx · 2024-11-05T02:25:50Z

text/0115-circuit-breaker.md

+
+```go
+type Settings struct {
+	Type                  string


How about adding more details for Type here?

This is a type of circuit breaker e.g. "pd_get_regions", "pd_tso", "tikv_copr". Maybe we can call it a Name?

Both Type and Name are ok to me and we can add some explanations in RFC.

rleungx · 2024-11-05T02:27:07Z

text/0115-circuit-breaker.md

+* Default: 10
+* Unit: seconds
+---
+* `min_qps_to_open`


Do we need a config for it or is it necessary?

I think so. Imagine we had only 1 request within error_rate_window which fails. At the end of window it will be evaluated at 100% error rate and open the circuit.

If "had only 1 request within error_rate_window", you do not have to enable the cb?

Usually circuit breaker tracks the number of failures within error window. I found it problematic to set absolute value given that it could vary a lot depending on qps hence I chose to go with error rate instead. But in this case we do need a protection from low qps cases or time of the day hence I propose minQPS. If we want to limit number of easy configurations to one session variables, then maybe we should consider to set it in absolute number of errors within window.

okJiang · 2024-11-05T06:41:52Z

text/0115-circuit-breaker.md

+* Unit: integer
+---
+* `cooldown_interval`
+* Defines how long to wait after circuit breaker is open before go to half-open state to send a probe request. This interval always equally jittered in the `[value/2, value]` interval.


Did I miss something? Why is it a range?

It is not a range. I just meant that the cooldown time will be jittered and with the actual time falling in the interval. Same as we jitter backoff time. Let me capture it more clearly here.

okJiang · 2024-11-05T06:47:14Z

text/0115-circuit-breaker.md

+
+#### System variables
+
+`tidb_cb_pd_metadata_error_rate_threshold_pct`:


Seems

pd_metadata is difficult to understand

error_rate is redundant with pct

How about renaming it to pd_client_cb_error_rate_threshold?

Do we expect to have a single circuit breaker covering all PD calls. I envisioned to have a dedicated circuit breaker for get regions calls. pd_client_get_regions_cb_error_rate_threshold?

pct meant to inidicate that the value is integer representing percent and not float representing the ratio.

If I remember correctly, the conclusion of the last meeting was that a single switch should control all interfaces.

Do you have any suggestions about this? @rleungx @nolouch

If we have a dedicated circuit breaker for get regions calls, do we need some variables to control other calls?

@okJiang I believe on last meeting we discussed which variables needs to be configured in real time by customer and which one could be hardcoded or configured at startup with toml file. My perception was that we want to allow configure error rate dynamically and everything else hardcoded or configured statically.

@rleungx I believe we need a separate circuit breaker instance for different calls: e.g. circuit breaker for region metdata should be separate to tikv coprocessor. They also needs to be enabled/disabled individually hence we need a separate variables for each call.

How many different circuit breakers we need could be decided later and we can follow same naming and granularity for configurations.

Do you mean we are only developing the region metadata part this time?🤔 @Tema

I'm open to add more. Let's brainstorm which one we want and see if they would need any different default settings. I think I mentioned before that some of other APIs requiring circuit breakers could be:

tikv to PD heartbeat (heatrbeat_cb_error_rate_threshold this prolly will be tikv.toml and not s sysvar)

tidb to each tikv requests ((tikv_client_cb_error_rate_threshold ))
Thoughts?

IMO, it is better to manage an interface with a variable. For the circuit breaker itself, different interfaces can share the same implementation and only use a different setting. Right now we can focus on the GetRegion call. /cc @okJiang

Tema

@rleungx Thanks for the review. I've asked some clarifying questions.

Tema · 2024-11-05T19:16:44Z

text/0115-circuit-breaker.md

+* Unit: integer
+---
+* `cooldown_interval`
+* Defines how long to wait after circuit breaker is open before go to half-open state to send a probe request. This interval always equally jittered in the `[value/2, value]` interval.


It is not a range. I just meant that the cooldown time will be jittered and with the actual time falling in the interval. Same as we jitter backoff time. Let me capture it more clearly here.

Tema · 2024-11-05T19:18:19Z

text/0115-circuit-breaker.md

+* Default: 10
+* Unit: seconds
+---
+* `min_qps_to_open`


I think so. Imagine we had only 1 request within error_rate_window which fails. At the end of window it will be evaluated at 100% error rate and open the circuit.

Tema · 2024-11-05T19:21:50Z

text/0115-circuit-breaker.md

+
+#### System variables
+
+`tidb_cb_pd_metadata_error_rate_threshold_pct`:


Do we expect to have a single circuit breaker covering all PD calls. I envisioned to have a dedicated circuit breaker for get regions calls. pd_client_get_regions_cb_error_rate_threshold?

pct meant to inidicate that the value is integer representing percent and not float representing the ratio.

Tema · 2024-11-05T19:22:59Z

text/0115-circuit-breaker.md

+
+```go
+type Settings struct {
+	Type                  string


This is a type of circuit breaker e.g. "pd_get_regions", "pd_tso", "tikv_copr". Maybe we can call it a Name?

nolouch · 2024-11-11T03:31:03Z

text/0115-circuit-breaker.md

+All configs described above will be encapsulated in a `Settings` struct with ability to change error rate threshold dynamically:
+
+```go
+type Settings struct {


Will this configuration be exported for the user? And what is the relationship between this and 'system variables'?

error rate threshold will be a system variable, and everything else is hardcoded or exposed in not documented toml client section.

nolouch · 2024-11-11T03:33:09Z

text/0115-circuit-breaker.md

+func (cb *CircuitBreaker[T]) Execute(run func() (T, error, boolean)) (T, error)
+```
+
+There is a third boolean parameter in the function argument above, in case the provided function doesn't return an error, but the caller still wants to treat it as a failure for the circuit breaker counting purposes (e.g. empty or corrupted result). 


Perhaps we can run a test where only the gRPC and timeout errors are treated as actual errors, which might be sufficient.

This is up to the place where we wire the circuit breaker in. Boolean param allows to configure it. It make sense to start with timeouts, but having ability to tweak it in the API let us tune it further.

rleungx · 2024-11-18T07:34:29Z

If there is no more comment, we can move on. /cc @niubell @nolouch @okJiang

Signed-off-by: artem_danilov <[email protected]>

Tema · 2024-11-19T21:38:31Z

I think I've addressed all opened comments in last commit

Renamed Settings.Type to Settings.Name
Clarified doc for cooldown_interval
Added missing Settings.MinQPS
Added docs for each Settings field

rleungx · 2024-12-12T09:18:52Z

@nolouch @okJiang @niubell Do you have more comments? If not, I think we can merge this PR.

Signed-off-by: rishabh_mittal <[email protected]>

* RFC:follower read cache Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * update review comments Signed-off-by: rishabh_mittal <[email protected]> * update review comments Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * add changes Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * incorporate review comments Signed-off-by: rishabh_mittal <[email protected]> * incorporate review comments Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * introduce client circuit breaker (#115) Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update text/0113-follower-read-cache.md Co-authored-by: cfzjywxk <[email protected]> Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * Update 0113-follower-read-cache.md Signed-off-by: rishabh_mittal <[email protected]> * review comments Signed-off-by: rishabh_mittal <[email protected]> --------- Signed-off-by: rishabh_mittal <[email protected]> Co-authored-by: Artem Danilov <[email protected]> Co-authored-by: cfzjywxk <[email protected]>

ti-chi-bot bot added the dco-signoff: no label Oct 31, 2024

Tema force-pushed the 0115-circuit-breaker branch from 5fabecb to 15631bf Compare October 31, 2024 00:06

ti-chi-bot bot added dco-signoff: yes and removed dco-signoff: no labels Oct 31, 2024

rleungx reviewed Oct 31, 2024

View reviewed changes

rleungx requested a review from nolouch October 31, 2024 02:59

ti-chi-bot bot added dco-signoff: no and removed dco-signoff: yes labels Oct 31, 2024

introduce client circuit breaker

eef6b58

Signed-off-by: artem_danilov <[email protected]>

Tema force-pushed the 0115-circuit-breaker branch from 300ef37 to eef6b58 Compare October 31, 2024 20:02

ti-chi-bot bot added dco-signoff: yes and removed dco-signoff: no labels Oct 31, 2024

rleungx reviewed Nov 5, 2024

View reviewed changes

okJiang reviewed Nov 5, 2024

View reviewed changes

rleungx mentioned this pull request Nov 5, 2024

Implement Circuit Breaker Pattern to Protect PD Leader tikv/pd#8678

Open

Tema commented Nov 5, 2024

View reviewed changes

nolouch reviewed Nov 11, 2024

View reviewed changes

rleungx requested review from okJiang and nolouch November 18, 2024 03:12

address comments

986a0dd

Signed-off-by: artem_danilov <[email protected]>

rleungx approved these changes Nov 25, 2024

View reviewed changes

Tema mentioned this pull request Nov 27, 2024

client: introduce circuit breaker for region calls tikv/pd#8856

Merged

nolouch approved these changes Dec 12, 2024

View reviewed changes

okJiang approved these changes Dec 12, 2024

View reviewed changes

rleungx merged commit 659ed1f into tikv:master Dec 12, 2024
2 checks passed

mittalrishabh pushed a commit to mittalrishabh/rfcs that referenced this pull request Dec 18, 2024

introduce client circuit breaker (tikv#115)

ff8c3de

Signed-off-by: rishabh_mittal <[email protected]>

mittalrishabh pushed a commit to mittalrishabh/rfcs that referenced this pull request Dec 18, 2024

introduce client circuit breaker (tikv#115)

060501a

Signed-off-by: rishabh_mittal <[email protected]>

mittalrishabh pushed a commit to mittalrishabh/rfcs that referenced this pull request Dec 20, 2024

introduce client circuit breaker (tikv#115)

ba4d37d

Signed-off-by: rishabh_mittal <[email protected]>

mittalrishabh pushed a commit to mittalrishabh/rfcs that referenced this pull request Dec 23, 2024

introduce client circuit breaker (tikv#115)

46c9080

Signed-off-by: rishabh_mittal <[email protected]>


		#### System variables

		`tidb_cb_pd_metadata_error_rate_threshold_pct`:

introduce client circuit breaker #115

introduce client circuit breaker #115

Conversation

Tema commented Oct 31, 2024 • edited Loading

Tema commented Oct 31, 2024

Tema commented Oct 31, 2024

Choose a reason for hiding this comment

rleungx commented Nov 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rleungx Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

okJiang Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tema left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rleungx commented Nov 18, 2024

Tema commented Nov 19, 2024

rleungx commented Dec 12, 2024 • edited Loading

Tema commented Oct 31, 2024 •

edited

Loading

rleungx Nov 5, 2024 •

edited

Loading

okJiang Nov 5, 2024 •

edited

Loading

rleungx commented Dec 12, 2024 •

edited

Loading