Use patch instead of update for GroupSnapshots, VolumeSnapshots, PVCs

[kaovilai]

Signed-off-by: Tiger Kaovilai [email protected]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
This PR cleans up some of the unit test reactor code, and eliminate update calls that I can see, fixing unit tests to accommodate the changes.
Previously update call was required simply cause unit test is borked. Patch calls was modifying original input causing compare failures.
Which issue(s) this PR fixes:

Fixes #748

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

This PR extend #876 work to later added update calls.

[k8s-ci-robot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kaovilai
Once this PR has been reviewed and has the lgtm label, please assign jsafrane for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @kaovilai. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jsafrane]

This is racy. Consider another controller / user that added a finalizer to an object when this loop runs. Since the PATCH removes indices, it will remove a wrong item.

Is there a way how to remove a value and not an index via Patch()? If not, then please stick to Update()

[kaovilai]

I believe there is.
One is to remove all then add all back in one call. Let me try that.

[kaovilai]

Fyi already have it working, just working on adding patch for pvc to framework_test reactor

[kaovilai]

If this is still considered racy, then will have to wait for json-patch/json-patch2#18 and will move back to update. lmk.

[kaovilai]

It probably still is racy.. hmm. but at least it external-snapshotter won't hang. User could see the finalizer they added isn't there. but at least this won't have the "wrong index" issue.

Another would be to use update but test it can get out of the "out of date, please apply again"

[jsafrane]

If I understand it correctly, the current version does not fix the race. It blindly removes all finalizers and add those that were known at the time the controller processed VolumeSnapshot / VolumeSnapshotContent.
It will again erase any finalizers added in parallel to the controller.

While it is a bandaid, it is better than nothing

No. You are breaking finalizers of someone else. This is not a good behavior.

I think the whole fear of "the object has been modified" error is unjustified. It tells the controller that it has been working with stale data. The controller should check what has changed and try again, if it's still applicable. Or it may discover that the work is not needed any longer. In this case, the VolumeSnapshot / Content should be re-queued with exp. backoff.

there needs to be tests that ensures update fail calls in external-snapshotter are recoverable within few seconds, not over 10 minutes which is what we have been seeing.

If that's true then this is the part that needs to be fixed. And not work around it using patch blindly everywhere. Can you reliably reproduce the issue, e.g. in an unit test? It should be easy to debug what's causing the delay then.

[kaovilai]

Yes we've been hitting this issue everywhere in our CI and other velero users have been hitting it as well.

[kaovilai]

		{
			name:             "2-4 - successful remove Snapshot finalizer after update conflict",
			initialSnapshots: newSnapshotArray("snap2-4", "snapuid2-4", "claim2-4", "", classSilver, "", &False, nil, nil, nil, false, true, nil),
			initialClaims:    newClaimArray("claim2-4", "pvc-uid2-4", "1Gi", "volume2-4", v1.ClaimBound, &classEmpty),
			test:             testRemoveSnapshotFinalizerAfterUpdateConflict,
			expectSuccess:    true,
			errors: []reactorError{
				{"update", "volumesnapshots", errors.NewConflict(crdv1.Resource("volumesnapshots"), "snap2-4", nil)},
			},
		},

Added this case here.
#1023 (review)

[kaovilai]

The requeue with new data shouldn't take more than a minute. We've seen the external-snapshotter controller stuck for 10minutes+ Maybe timeout/backoff used needs changing.

[kaovilai]

reliably reproduce the issue

50% of the time prior to #876
After 876 it's much improved but added update calls after that OCP QE noted still cause issues sometimes.

[shubham-pampattiwar]

Thank you @kaovilai !!

[kaovilai]

Closing due to other priorities as the sense is this isn't going to be approved as is and to see if #876 recently cherrypicked to our nightlies would be sufficient. Feel free to takeover.

Test improvements are moved to #1024