NO-ISSUE: OTA-1605 Create e2e test in CVO repo #1249
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: JianLi-RH The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
JianLi-RH
force-pushed
the
implement_client_oc
branch
4 times, most recently
from
a6a8ada to
37caff4
Compare
JianLi-RH
changed the title
JianLi-RH
force-pushed
the
implement_client_oc
branch
4 times, most recently
from
a79646a to
d944000
Compare
|
Please note that
|
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
d944000 to
067cd4b
Compare
JianLi-RH
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@JianLi-RH: This pull request explicitly references no jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Tested on my local machine: Run the case: |
JianLi-RH
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
067cd4b to
bac7793
Compare
|
/cc |
|
/cc |
test/README.md
Outdated
There was a problem hiding this comment.
We have https://github.com/openshift/cluster-version-operator/blob/main/cmd/cluster-version-operator-tests/README.md already.
That folder is the entry of the test extension. I feel it fits better to hold the README.md there.
We could merge the useful part into that file.
test/README.md
Outdated
| # Cluster Version Operator Tests | ||
|
|
||
| We hold all e2e tests and related common methods in this folder. | ||
| Test script are using client-go, here is the document: https://pkg.go.dev/k8s.io/client-go |
There was a problem hiding this comment.
I do not think we need to mention the dependency of the test, unless there are something special for this particular one: client-go.
There was a problem hiding this comment.
This description is helpful for new contributors who are unfamiliar with client-go.
test/README.md
Outdated
| - Install golang, the version should match it in go.mod. Here is the how-to doc https://go.dev/doc/install | ||
| - Have the environment variable `KUBECONFIG` set pointing to your cluster. |
There was a problem hiding this comment.
- It should be fine if the installed GoLang version > the claimed version in go.mod.
- How the test loads KUBECONFIG is up to the implementation.
In general, we do not need to document those things. If they are not installed/configured correctly, our test should just error out with the necessary hint.
There was a problem hiding this comment.
I followed the README in https://github.com/openshift/openshift-tests-private, I think not all user can debug issues when the environment are not setup.
test/README.md
Outdated
| ## Compile the executable binary | ||
| In root folder, run below command to build executable binary: | ||
| ``` | ||
| $ make update |
There was a problem hiding this comment.
The build target sounds fitting better.
| $ make update | |
| $ make build |
There was a problem hiding this comment.
Because I see make update do more work than make build, and sometimes it will cause issues when there is no .openshift-tests-extension/openshift_payload_cluster-version-operator.json
update: build
hack/update-test-metadata.sh
.PHONY: update
I will replace then command.
test/README.md
Outdated
| - Install golang, the version should match it in go.mod. Here is the how-to doc https://go.dev/doc/install | ||
| - Have the environment variable `KUBECONFIG` set pointing to your cluster. | ||
|
|
||
| ## Compile the executable binary |
There was a problem hiding this comment.
| ## Compile the executable binary | |
| ## Build the executable binary |
|
|
||
| _, err = oc.RbacV1().ClusterRoleBindings().Get(context.TODO(), "default-account-openshift-machine-config-operator", metav1.GetOptions{}) | ||
| o.Expect(err).To(o.HaveOccurred(), "ClusterRoleBinding default-account-openshift-machine-config-operator was accidentally installed") | ||
|
|
test/cvo/cvo.go
Outdated
| g.By("Validate resource with 'release.openshift.io/delete: \"true\"' annotation is not installed") | ||
|
|
||
| _, err := oc.CoreV1().Services("openshift-cloud-credential-operator").Get(context.TODO(), "controller-manager-service", metav1.GetOptions{}) | ||
| o.Expect(err).To(o.HaveOccurred(), "Service controller-manager-service was accidentally installed") |
There was a problem hiding this comment.
We need to check if the error is notFound, something like
o.Expect(kerrors.IsNotFound(err)).To(o.BeTrue(), "The NotFound error should occur")
with
kerrors "k8s.io/apimachinery/pkg/api/errors"
| bin/ | ||
| .DS_Store | ||
| _output | ||
| .openshift-tests-extension |
There was a problem hiding this comment.
I am wondering why the file showed up.
There was a problem hiding this comment.
I added this line otherwise we need to update the file every time when we add/remove tests
There was a problem hiding this comment.
The verify-update job was failing with:
Missing Tests:
* [Jira:Cluster Version Operator] cluster-version-operator-tests should support passing tests
Error: missing tests, if you've renamed tests you must add their names to OriginalName, or mark them obsolete
An updated readme/docs is to be expected for onboarding folks, and this was a nice reminder for me to do that; however, please do not disregard such errors by "bypassing" them. If so, please create a distinct commit with the "what" and "why" for reviewers and future reference.
There was a problem hiding this comment.
Why is it deleted?
From cfab71d i feel it needs an update tho.
There was a problem hiding this comment.
It will be created automatically when we run make update, I don't see any reason that we need to keep this file in repo.
cc @DavidHurta WDYT?
There was a problem hiding this comment.
Not doing a thorough review at the moment, I have noticed my ping.
The file is required.
Update - Metadata Validation
Component owners will be responsible for implementing the extension interface. To prevent common mistakes and ensure conformance with the evolution of their implementation, make (or similar build system) must run update [--component product:type:component] [--basedir ] after the extension binary is built.
The update verb will create or update files under hack/.openshift-tests-exension/product/type/component/*, by default (basedir defaults to hack/.openshift-tests-extension). If an incompatible change is introduced from the prior invocation of update (e.g. changing a test name without preserving the original), update will raise an error which the component owner must correct before committing their change in git.
The content of the files stored under .openshift-tests-extension is subject to change and wholly defined by the github.com/openshift-eng/openshift-tests-extension module. Individual component owners should not modify files under this path except through invocations of update.
However, this was a nice exercise. I will improve the make update job to explicitly expect the file (in this PR, this was bypassed by modifying the gitignore file). I will also reference this in a respective docs/readme file.
test/cvo/cvo.go
Outdated
| var _ = Describe("[Jira:Cluster Version Operator] cluster-version-operator-tests", func() { | ||
| It("should support passing tests", func() { | ||
| Expect(true).To(BeTrue()) | ||
| var _ = g.Describe("[Jira:Cluster Version Operator] cluster-version-operator-tests", g.Ordered, g.Label("cvo"), func() { |
There was a problem hiding this comment.
| var _ = g.Describe("[Jira:Cluster Version Operator] cluster-version-operator-tests", g.Ordered, g.Label("cvo"), func() { | |
| var _ = g.Describe("[Jira:Cluster Version Operator] The cluster version operator", g.Ordered, g.Label("cvo"), func() { |
Usually the specs on the nodes can make a sentence.
https://onsi.github.io/ginkgo/#writing-specs
JianLi-RH
force-pushed
the
implement_client_oc
branch
from
bac7793 to
51facb3
Compare
|
After modification, the case is still valid: |
|
@JianLi-RH: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
4 participants
In this PR, finished first e2e test case. Related case can be found in: https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-42543
Tested it locally:
Here is the old case in openshift-tests-private: https://github.com/openshift/openshift-tests-private/blob/f1f32ff1e9eeef4cb2cb4ae6549daa9306c6616a/test/extended/ota/cvo/cvo.go#L716-L759
There is a disadvantage here when I use client-go, it seems client-go does not support all Openshift features, for example it could not extract all manifest and it although support methods to get all resource types but it could not use these types to get resources. So in these PR I don't check all manifests but check some special objects.
/cc @wking @DavidHurta @PratikMahajan @hongkailiu @jhou1 @dis016 @jiajliu