Retry reloading Visibility Manager search attributes cache when cold #8299
+86
−55
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rodrigozhou
changed the title
rodrigozhou
changed the title
yiminc
reviewed
Sep 8, 2025
| saCache.expireOn = now.Add(cacheRefreshIfUnavailableInterval) | ||
| default: | ||
| return saCache, err | ||
| if saCache.dbVersion == 0 { |
There was a problem hiding this comment.
If cache is cold, 1) default refresh interval should be much shorter, 1s or even shorter than that. 2) we should retry on all errors if cache is code, not just Unavailable error, like resource exhausted or timeout errors.
There was a problem hiding this comment.
Reduced the interval to 1 second.
If there are any other error besides NotFound or Unavailable, it will retry immediately no matter if the cache is cold or not. This is the existing logic, and I didn't change.
yiminc
approved these changes
Sep 8, 2025
rodrigozhou
force-pushed
the
rodrigozhou/vis-sa-cache
branch
from
b8fec1e to
180f321
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
If the visibility manager's search attribute cache is cold (ie., never loaded yet), and reading cluster metadata failed (
Unavailableerror), then retry randomly within 5 seconds instead of the default 20 seconds.Why?
If the cache is cold and there are custom search attributes, then reading or writing workflows with custom search attributes would fail for 20 seconds. Retrying within 5 seconds to minimize the impact.
How did you test it?
Potential risks