Repository navigation
[release-0.33] Return NotFound to WatchList clients when a shard does not serve the resource - #4547
Conversation
…resource A watch for a resource that is not served on this shard is answered with an empty watch that yields no events and stays open, so that a wildcard consumer aggregating across shards is not wedged by a shard that simply holds none of these objects. A WatchList client cannot use such a stream. It treats the watch as its initial list and waits for the server to close that list with an "initial-events-end" bookmark before it considers its store synced. An empty watch sends no such bookmark, and never errors either, so nothing makes the client relist — the comment's "objects that later appear on this shard are picked up on the next relist" never happens, because there is no next relist. The client waits forever: reflector.go:327: awaiting required bookmark event for initial events stream, no events received for 10.004121875s That is reachable in a normal deployment. A controller consuming an APIExport through its virtual workspace starts before any workspace has bound the export, so the endpoint slice is still empty; when the first binding appears, the controller connects and establishes a watch for a resource this shard does not serve yet. Its informers never sync and it never reconciles anything, with no error anywhere to explain it. Surface the NotFound to those clients instead: they retry, and pick the resource up once the shard serves it. Ordinary watches keep the empty watch and the behaviour that was added for them. An alternative would be for the empty watch to emit the initial-events-end bookmark itself, which would let a WatchList client sync against a shard that holds nothing. That needs a resourceVersion to report as synced-at, and I did not want to invent one; if there is a right value to use here, that is the better fix. Signed-off-by: Waldemar Kindler <waldemar.kindler@think-ahead.tech> (cherry picked from commit e99a021) Signed-off-by: mjudeikis-bot <263128906+mjudeikis-bot@users.noreply.github.com>
|
Hi @mjudeikis-bot. Thanks for your PR. I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
|
LGTM label has been added. DetailsGit tree hash: 62590d427779be8d3579e15351283e4d12974074 |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ntnn The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/kind bug |
Summary
Backport of #4384 to release-0.33, superseding #4546. Same change as #4546, with the one fix it needed: in
TestWatchNotFoundOnShard,newStoragereturns two values on release-0.33 (main returns three), so the call is nowstorage, _ := newStorage(...). #4546 failed to compile its test package, which is what brokepull-kcp-test-unitandpull-kcp-lint.Requested in #4546 (comment)
Conflicts: none in the cherry-pick itself; only the test-helper signature difference above.
Tested
go vet ./pkg/forwardingregistry/in the virtual-workspace-framework module: cleango test -race ./pkg/forwardingregistry/: passesgolangci-lint run ./...in the virtual-workspace-framework module (repo config): 0 issues