Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: reduce acceptance test flakiness #632

Merged
merged 7 commits into from
Nov 7, 2024
Merged

Conversation

rjaegers
Copy link
Member

@rjaegers rjaegers commented Nov 6, 2024

Pull Request

Description of changes

This PR solves a few issues that reduce the flakiness of acceptance tests. The most apparent one being waiting on the Codespace to be started before running the tests.

Checklist

  • I have followed the contribution guidelines for this repository
  • I have added tests for new behavior, and have not broken any existing tests
  • I have added or updated relevant documentation
  • I have verified that all added components are accounted for in the SBOM

@rjaegers rjaegers requested a review from a team as a code owner November 6, 2024 12:48
Copy link
Contributor

github-actions bot commented Nov 6, 2024

Compressed layer size comparison

Comparing ghcr.io/philips-software/amp-devcontainer-cpp:latest to ghcr.io/philips-software/amp-devcontainer-cpp@sha256:d40ce0390cb57433322e36c381b55b1e7a93735c8a308db723c3f853db8c405c

OS/Platform Previous Size Current Size Delta
linux/amd64 650.42M 650.48M 57.71K (+0.01%)
linux/arm64 643.10M 643.16M 62.00K (+0.01%)

Copy link
Contributor

github-actions bot commented Nov 6, 2024

Compressed layer size comparison

Comparing ghcr.io/philips-software/amp-devcontainer-rust:latest to ghcr.io/philips-software/amp-devcontainer-rust@sha256:201171bda9bb1c8f8fa6418122063f7ec435469aa9ae1c70a0855945f2a77e84

OS/Platform Previous Size Current Size Delta
linux/amd64 433.00M 459.82M 26.82M (+6.19%)
linux/arm64 572.52M 599.31M 26.80M (+4.68%)

Copy link
Contributor

github-actions bot commented Nov 6, 2024

🦙 MegaLinter status: ✅ SUCCESS

Descriptor Linter Files Fixed Errors Elapsed time
✅ ACTION actionlint 16 0 0.05s
✅ DOCKERFILE hadolint 2 0 0.37s
✅ GHERKIN gherkin-lint 2 0 0.46s
✅ JSON npm-package-json-lint yes no 0.38s
✅ JSON prettier 15 1 0 0.65s
✅ JSON v8r 14 0 16.78s
✅ MARKDOWN markdownlint 8 0 0 0.92s
✅ MARKDOWN markdown-table-formatter 8 0 0 0.23s
✅ REPOSITORY checkov yes no 17.65s
✅ REPOSITORY gitleaks yes no 0.28s
✅ REPOSITORY git_diff yes no 0.01s
✅ REPOSITORY grype yes no 15.56s
✅ REPOSITORY secretlint yes no 1.43s
✅ REPOSITORY syft yes no 0.37s
✅ REPOSITORY trivy yes no 7.87s
✅ REPOSITORY trivy-sbom yes no 0.12s
✅ REPOSITORY trufflehog yes no 1.94s
✅ SPELL lychee 57 0 2.73s
✅ YAML prettier 21 0 0 1.07s
✅ YAML v8r 21 0 14.76s
✅ YAML yamllint 21 0 0.39s

See detailed report in MegaLinter reports

You could have the same capabilities but better runtime performances if you use a MegaLinter flavor:

MegaLinter is graciously provided by OX Security

Copy link
Contributor

github-actions bot commented Nov 6, 2024

Test Results

 2 files  ±0   2 suites  ±0   1m 21s ⏱️ -2s
29 tests ±0  29 ✅ ±0  0 💤 ±0  0 ❌ ±0 
31 runs  ±0  31 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 7067274. ± Comparison against base commit 92ea3e3.

♻️ This comment has been updated with latest results.

@rjaegers rjaegers changed the title test: make plugin Locator match exact test: reduce acceptance test flakyness Nov 7, 2024
@magi-arun magi-arun requested a review from EkelmansPh November 7, 2024 08:16
Copy link

sonarqubecloud bot commented Nov 7, 2024

@rjaegers rjaegers changed the title test: reduce acceptance test flakyness test: reduce acceptance test flakiness Nov 7, 2024
@rjaegers rjaegers added this pull request to the merge queue Nov 7, 2024
Merged via the queue into main with commit 396057d Nov 7, 2024
27 checks passed
@rjaegers rjaegers deleted the feature/reduce-test-flakyness branch November 7, 2024 11:15
Copy link
Contributor

github-actions bot commented Nov 7, 2024

Pull Request Report (#632)

Static measures

Description Value
Number of added lines 36
Number of deleted lines 9
Number of changed files 3
Number of commits 7
Number of reviews 1
Number of comments (w/o review comments) 5
Number of reviews that contains a comment to resolve 0
Number of reviews that requested a change from the author 0
Number of reviews that approved the Pull Request 1
Get the total number of participants of a Pull Request 4

Time related measures

Description Value
PR lead time (from creation to close of PR) 22.5 Hours
Time that was spend on the branch before the PR was created 28 Sec
Time that was spend on the branch before the PR was merged 22.5 Hours
Time to merge after last review 7.4 Min

Status check related measures

Description Value
Total runtime for last status check run (Workflow for PR) 19.3 Min
Total time spend in last status check run on PR 2.5 Hours

Copy link
Contributor

github-actions bot commented Nov 7, 2024

🎉 Hooray! The changes in this pull request went live with the release of v5.5.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants