Run e2e-tests in CI #1210
Open
Run e2e-tests in CI #1210
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
denisonbarbosa
force-pushed
the
e2e-tests-in-ci
branch
from
f088faa to
e081a57
Compare
adombeck
force-pushed
the
e2e-tests-in-ci
branch
from
1e8856e to
4784536
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1210 +/- ##
==========================================
+ Coverage 80.57% 87.64% +7.06%
==========================================
Files 20 91 +71
Lines 978 6231 +5253
Branches 0 111 +111
==========================================
+ Hits 788 5461 +4673
- Misses 190 714 +524
- Partials 0 56 +56 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
denisonbarbosa
force-pushed
the
e2e-tests-in-ci
branch
3 times, most recently
from
5fd0acc to
4201248
Compare
adombeck
commented
Jan 28, 2026
adombeck
force-pushed
the
e2e-tests-in-ci
branch
from
6e51dc9 to
8cfdaef
Compare
adombeck
commented
Jan 28, 2026
.github/workflows/e2e-tests.yaml
Outdated
Comment on lines
75
to
83
| - name: Clean previous cache | ||
| if: always() && steps.provision-vm.outcome == 'success' | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| GH_REPO: ${{ github.repository }} | ||
| run: | | ||
| set -eu | ||
| gh cache delete e2e-runner-${{ matrix.release }} || true | ||
| echo "Previous cache (if any) deleted" |
Contributor
Author
There was a problem hiding this comment.
Can we add a comment here explaining why it's necessary to delete the cache even though we had a cache miss above?
adombeck
force-pushed
the
e2e-tests-in-ci
branch
2 times, most recently
from
61c7d29 to
9f93b5a
Compare
This makes our life easier when trying to upload the test outputs as artifacts on CI.
The syntax used in this execution was flawed.
This is a QoL change to help us fine tune the artifacts in CI.
We were using only the base name to create the image file, which could cause conflicts if we ever try to setup both releases.
The 'await' command on virsh is not available on Noble, which is our runner in github, so we need to use another way to check for the domain status.
Update the commands to also run `sudo apt-get update` and add some dependencies that were missing from the lists
This workflow downloads an Ubuntu image, sets up some of the aplications and configurations that we will need in the VM in order to be able to run the E2E tests and then caches it for later use to save us some time when running the tests. Co-authored-by: Adrian Dombeck <[email protected]>
fix output artifact naming ci/e2e-tests: Use env vars instead of config file The config file is useful when running the tests locally, but it only sets environment variables, so in the CI it's simpler to just set the environment variables directly. ci/e2e-tests: Fix indentation
This caused the provisioning to fail
We were padding the generated code with trailing 0s, but we should have been padding with leading 0s. This commit fixes that.
The code is generated according to the current time and is valid for 30 seconds. This means that if we generate the code just before the time window changes, it might be invalid by the time we use it. To avoid this, we make sure the time is safely within a new window before generating the code.
We see permission denied errors when trying to start the VM both in CI and locally because AppArmor denies access to the qcow2 image (curiously, starting the VM by restoring a running snapshot works). We should investigate the issue and report it upstream. In the meantime, let's work around it by disabling AppArmor confinement for the e2e-test VM. That should be fine since we're not doing anything in the VM which would make them more vulnerable than the host.
We recently changed the error message that is shown when the broker fails to reach the provider, so we need to update this check on the tests as well.
Sometimes machinectl login hangs and it causes the tests to fail. Let's try reopening it a couple of times (in case of failure) to see if we are able to get more consistent behavior
systemd only supports sending the journal via VSOCK since v256. In the GitHub CI the newest Ubuntu release we can use on the runners (both GitHub-hosted or self-hosted) is 24.04, which has systemd 255.
Running the tests is quite resource intensive, so let's switch to using our self-hosted runners to see if we can get some improvements in performance and stability.
It doesn't only "Build and Cache Ubuntu VM", it runs the e2e-tests and does everything required to do so.
On GitHub Runners, the VM can't connect to the host via TCP. This commit instead makes the host connect to the VM to retrieve the journal.
On the self-hosted runners, the tests fail because the SSH connection to the VM fails. Also, downloading the VM image from the artifacts takes 12 minutes instead of 20 seconds. This reverts commit 4201248.
The tests are also useful when targeting other branches than main.
There are not many files in the repo which don't affect the e2e-tests (once we run them with authd and broker artifacts built from the branch) but it's still worthwhile to add include/exclude paths because the tests are very resource intensive.
To avoid duplication between the msentraid and google e2e-tests.
So that we automatically re-provision the VM image if anything changed in the relevant files.
It's too much output
It's not only testing authd-oidc-brokers but also authd.
We ignored errors during cloud-init and happily continued provisioning a broken VM.
Should save some time and bandwidth.
The file is already executable
adombeck
force-pushed
the
e2e-tests-in-ci
branch
from
9f93b5a to
0e7717f
Compare
Contributor
Author
|
@denisonbarbosa I rebased and squashed the fixup commits. We're good to merge from my side. |
Contributor
Author
|
@denisonbarbosa I now see that there are still some |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Run our new e2e-tests in the GitHub CI.
UDENG-7426