Investigate incorrect CI failure reporting #3952
Open
+51
−28
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JonasKunz
added
the
ci:agent-integration
JonasKunz
commented
Jan 24, 2025
...gration-tests/src/test/java/co/elastic/apm/servlet/JakartaeeTomcatWithSecurityManagerIT.java
Show resolved
Hide resolved
SylvainJuge
previously approved these changes
Jan 24, 2025
...apm-jdbc-plugin/src/test/java/co/elastic/apm/agent/jdbc/AbstractJdbcInstrumentationTest.java
Show resolved
Hide resolved
apm-agent-plugins/apm-jdbc-plugin/src/test/java/co/elastic/apm/agent/jdbc/JdbcDbIT.java
Show resolved
Hide resolved
SylvainJuge
approved these changes
Jan 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the problems with CI sometimes not correctly reporting test failures as actually failed.
The root cause for this problem was a old
maven-surefire-plugincombined with parameterized tests.E.g. our
TomcatITtest is parameterized, the parameter is the name of the tomcat docker image to test.#3940 removes an image which doesn't exist anymore, thus the tests should have been failing but were not.
The reason was that the used version
maven-surefire-pluginwas naming all runs of parametrized tests with the same name - even when executed with different parameters. E.g. All test runs for different tomcat images were named justTomcatIT.testAllScenariosin the resulting test report XML file.Subsequently, if at least one of the runs was successful (aka at least one image works) the tests would not count as failure, but as a single test being flaky. Flaky tests do not cause maven to report the tests as failed.
The update of the
maven-surefire-pluginfixes this, because it assigns unique names to parametrized tests based on the parameters.