[jUnit5] [jupiter-engine] removing the "suite" label that AllureJunitPlatform automatically applies causes unexpected behavior

[grays0da]

Description of unexpected behavior

I encountered the issue described allure-framework/allure-java#1028 and applied the fix described in the issues comments. Upon doing so the report began grouping some of the tests under their package name instead of the "parentSuite", "suite", and "subSuite" labels or any "epic", "feature", or "story" labels defined in the tests BeforeEach method. Additionally the tests grouped under their package name DO NOT show any steps, parameters, links, or other programmatically defined metadata (they do have the correct pass/fail status). That data IS still present in the report but the user must click on the "retries" tab and click the entry there to bring up the actual test run data with all associated metadata.

These tests are using jUnit5's @TestTemplate annotation. They are being discovered, launched, and executed by the default jUnit 5 org.junit.platform.launcher.Launcher created from the LauncherFactory.create() method. The AllureJunitPlatform object is being instantiated and attached as a TestExecutionListener through the launcher.execute(TestPlan testPlan, TestExecutionListener... listeners) method. They are selenium tests which may be executed locally or on a remote selenium grid instance. This does not effect the behavior I am reporting.

The test runs which are reported under their package name are seemingly random. They do not seem to depend on parallel execution conditions (parallel or single threaded produce the same random results). They do not depend on what site is visited or if the test is executed locally or remotely. They change every run.

Additionally (and potentially unrelated) the generated reports do not show the Behaviors or Packages sections.

What Allure Integration are you using?

allure-junit5

What version of Allure Integration you are using?

2.29.1

What version of Allure Report you are using?

I'm unsure. I am generating the reports using the allure-generator package https://mvnrepository.com/artifact/io.qameta.allure/allure-generator version 2.32.2. I have also tried with the allure command line utility version 2.32.0 with the same results.

[grays0da]

Turns out that a workaround I was doing for jUnit issue junit-team/junit-framework#3108 was causing this issue. In configuring the fixed parallel execution I was setting junit.jupiter.execution.parallel.config.fixed.parallelism equal to <# threads> / 2 and junit.jupiter.execution.parallel.config.fixed.max-pool-size equal to <# threads> as a workaround for ForkJoinPool's greedy test execution. I'm guessing that this caused not enough threads to be available which caused the behavior reported in this post.

[grays0da]

never mind that wasn't it either