Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create monitor for alarming on cache drift #167

Open
jyotimahapatra opened this issue Oct 19, 2020 · 1 comment
Open

Create monitor for alarming on cache drift #167

jyotimahapatra opened this issue Oct 19, 2020 · 1 comment

Comments

@jyotimahapatra
Copy link
Contributor

While deploying xdsrelay in staging environment for a few critical services, we realized that the cache for eds had a drift.
The cache drift perhaps happened due to a bug. We need a way in xdsrelay to compare cache entried periodically and alarm when they dont match. The xdsrelay operators need a way to at least restart the server and reset the cache in order to mitigate when this happens.
There could always be bugs during initial launch, and such a method can keep the system safe. This feature should be based on a bootstrap config so that it can be enabled only when configured. It also needs a way to transform xdsrelay keys with upstream control plane keys.

@jyotimahapatra
Copy link
Contributor Author

@jessicayuen We need some kind of monitor to make sure we alarm when cache drifts. We can make such a mechanism inhouse or think about this feature in the open from the outset, since this could be a helpful feature for operators to safely rollout the service. wdyt?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant