HTTPBackendRef RequestMirror Conformance Test

[dprotaso]

What type of PR is this?

/kind cleanup
/kind test
/area conformance

What this PR does / why we need it

There's currently no conformance test for this feature which is 'Extended'

Additional notes

I dropped path logic in filter so we can re-use the same test for the backend

NONE

[dprotaso]

/kind cleanup
/kind test
/area conformance

[dprotaso]

adding approvers
/assign @arkodg @mlavacca @sunjayBhatia

adding reviewers
/cc @LiorLieberman @michaelbeaumont @Xunzhuo

[dprotaso]

Conformance reviewers/approvers - can I get review please @arkodg @mlavacca @sunjayBhatia @michaelbeaumont @LiorLieberman @Xunzhuo

[dprotaso]

I've added back all the test cases for the regular filter. The backend filter has a subset of these cases.

[arkodg]

has this been tested against any implementation ? afaik this isnt easily possible on envoy based implementations

[dprotaso]

has this been tested against any implementation ?

I ran these tests against Istio. The backendRef variant fails as expected because they don't support that feature.

[arkodg]

imo lets add conformance tests if at least 1 implementation supports this case.

[robscott]

imo lets add conformance tests if at least 1 implementation supports this case.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

[dprotaso]

imo lets add conformance tests if at least 1 implementation supports this case.

But then an implementation can't advertise the support this feature on the GatewayClass. They'll have to wait for a new gateway api release to do this.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

Then why is this functionality in the API? If we have the shape of feature in our API we can write conformance for it

[dprotaso]

Also the test is identical to the pre-split filter - so for this case I wouldn't be concerned with stability of the conformance test.

[mlavacca]

imo lets add conformance tests if at least 1 implementation supports this case.

But then an implementation can't advertise the support this feature on the GatewayClass. They'll have to wait for a new gateway api release to do this.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

Then why is this functionality in the API? If we have the shape of feature in our API we can write conformance for it

I +1 what @arkodg and @robscott said about waiting to have at least one implementation on board with this feature. In my opinion, conformance tests are not intended to shape the API, but instead to validate API implementations. Regardless of the nature of the test (this test could be straightforward, as it shares the same logic of the requestMirror feature), we should wait for at least one implementation to properly run the tests, otherwise, the risk of having tests which are just a bunch of untested code is pretty high (I'm not saying it's the case of this specific PR, my concern is about the approach we want to have).

[mlavacca]

In a backlog grooming session, we went through this issue and discussed the need to define the status of conformance tests (experimental/GA). Shane is going to take a look into it.

xref: #3009

/assign @shaneutt

[shaneutt]

@dprotaso Thanks for adding this!

In general, I think this looks good:

/approve

However, one thing that does give me some pause: We haven't really seen any support from other implementations saying that this test makes sense and works for them?

/hold

I can see at least two ways we can go about this. We could leave this PR on hold and do outreach, try to get some others to utilize it. However, we might need some kind of "test is experimental" flag so as to allow a soak period for new tests in lieu of immediate feedback from implementations? For the latter, we have a bit of a precedent in that we used an +experimental Go build tag for several months to stage in the version 2 of our conformance test suite (the one that introduced conformance reports). I'm not necessarily advocating for anything in particular yet, would like to know your thoughts @dprotaso ?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprotaso, shaneutt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [shaneutt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

@dprotaso: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gateway-api-crds-validation-1	eb2197f	link	true	/test pull-gateway-api-crds-validation-1
pull-gateway-api-crds-validation-5	eb2197f	link	true	/test pull-gateway-api-crds-validation-5
pull-gateway-api-crds-validation-3	eb2197f	link	true	/test pull-gateway-api-crds-validation-3
pull-gateway-api-crds-validation-4	eb2197f	link	true	/test pull-gateway-api-crds-validation-4
pull-gateway-api-crds-validation-2	eb2197f	link	true	/test pull-gateway-api-crds-validation-2

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[candita]

ping @dprotaso