rls: only reset backoff on recovery from TRANSIENT_FAILURE

[ulascansenturk]

Fix control channel connectivity monitoring to track TRANSIENT_FAILURE state explicitly. Only reset backoff timers when transitioning from TRANSIENT_FAILURE to READY, not for benign state changes like READY → IDLE → READY.

RELEASE NOTES: N/A

Fixes #8693

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.22%. Comparing base (50c6321) to head (ed5ab2c).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #8720   +/-   ##
=======================================
  Coverage   83.21%   83.22%           
=======================================
  Files         419      419           
  Lines       32427    32437   +10     
=======================================
+ Hits        26985    26995   +10     
- Misses       4054     4056    +2     
+ Partials     1388     1386    -2     

Files with missing lines	Coverage Δ
balancer/rls/control_channel.go	88.23% <100.00%> (+1.27%)	⬆️

... and 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[eshitachandwani]

I think this comment can be improved and made a little more explicit for ease of users.

[eshitachandwani]

I dont think adding time.Sleep() to wait for state changes is a good idea, since it can make tests flake if sometimes the state transitions take longer, we should look for better guarantees to make sure the states have transitioned.
cc : @easwars

[Copilot]

Pull request overview

This PR fixes control channel connectivity monitoring in the RLS balancer to only reset backoff timers when genuinely recovering from a TRANSIENT_FAILURE state, not during benign state changes like READY → IDLE → READY.

• Adds explicit tracking of TRANSIENT_FAILURE state with a boolean flag
• Updates callback invocation logic to only trigger after recovery from TRANSIENT_FAILURE
• Adds comprehensive test coverage for various state transition scenarios

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
balancer/rls/control_channel.go	Implements TRANSIENT_FAILURE tracking with a boolean flag, adds nil check for callback, and includes test synchronization channel
balancer/rls/control_channel_test.go	Adds comprehensive test cases covering different connectivity state transition scenarios

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[easwars]

I have we have test-only code/hooks in some parts of the code. But it would be nice to avoid these.

[easwars]

FYI: This is the exact text that describes the expected behavior:

The policy will monitor the state of the control plane channel. When the state transitions to TRANSIENT_FAILURE, it will record that transition, and the next time it transitions to READY, the policy will iterate through the cache to reset the backoff timeouts in all cache entries. Specifically, this means that it will reset the backoff state and cancel the pending backoff timer. Note that when cancelling the backoff timer, just like when the backoff timer fires normally, a new picker is returned to the channel, to force it to re-process any wait-for-ready RPCs that may still be queued if we failed them while we were in backoff. However, we should optimize this case by returning only one new picker, regardless of how many backoff timers are cancelled.

[easwars]

Based on the above text, we don't even have to wait for the first time the control channel goes READY. This means, that we can simplify the code quite a bit and not even have a control channel connectivity state monitoring goroutine. All we need is the following:

• Continue to subscribe to connectivity state changes as we do today when we create the RLS control channel:

  grpc-go/balancer/rls/control_channel.go

  Line 91 in cdbafd3

  ctrlCh.unsubscribe = internal.SubscribeToConnectivityStateChanges.(func(cc *grpc.ClientConn, s grpcsync.Subscriber) func())(ctrlCh.cc, ctrlCh)

• In the implementation of the grpcsync.Subscriber interface, we currently push the received connectivity state update on to an unbounded buffer here:

  grpc-go/balancer/rls/control_channel.go

  Line 104 in cdbafd3

  cc.connectivityStateCh.Put(st)

• The above buffer is read from the for loop in the monitoring goroutine here:

  grpc-go/balancer/rls/control_channel.go

  Line 177 in cdbafd3

  for s, ok := <-cc.connectivityStateCh.Get(); s != connectivity.Ready; s, ok = <-cc.connectivityStateCh.Get() {

• Instead, what we can do is:

func (cc *controlChannel) OnMessage(msg any) {
	st, ok := msg.(connectivity.State)
	if !ok {
		panic(fmt.Sprintf("Unexpected message type %T , wanted connectectivity.State type", msg))
	}

    - If new connectivity state is READY, and we have previously seen TRANSIENT_FAILURE:
      - set the boolean for tracking previously seen TRANSIENT_FAILURE to false
      - reset backoffs by invoking the `backToReadyFunc`
    - else if new connectivity state is TRANSIENT_FAILURE
      - set the boolean for tracking previously seen TRANSIENT_FAILURE to true
    - else
      - do nothing 
}

The above if-elseif-else can also be implemented as a switch and the linter might complain if that is not the case.