tidb logs are flooded with "unable to find a store with given labels"

[mittalrishabh]

Issue
We see a LOT of such logs in our clusters. 2.5M logs in 1hr across our 180 tidb pods.

[2024/06/13 22:39:50.475 +00:00] [WARN] [region_request.go:661] ["unable to find a store with given labels"] [region=2599577092] [stale-read=true] [labels="[{"key":"zone","value":"us-east-1a"}]"]

What we expect
It is spamming the logs. We would like to make it debug only if it is not a critical message.

It seems like it happens when a request is tried on all replicas and is failed. We are seeing 4 request/sec/tidb (2.5M/180/3600) failing on all replicas. I think it is an expected behavior in distributed systems.

[crazycs520]

This log has been removed in the master branch since replica selector v2 was introduced by https://github.com/tikv/client-go/pull/1265/files#diff-0c36d07fbd1fc71c850ff57d323fe12bdf7f2c053ef71b7fec5bb44358f1070dL1003