Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gradle retry plugin: Unable to handle failed test that has a random unique name per iteration #5633

Open
yuliiamir opened this issue Apr 10, 2024 · 1 comment

Comments

@yuliiamir
Copy link
Collaborator

yuliiamir commented Apr 10, 2024

The issue occurs when the parameterized Spock method with a random unique name for each iteration fails.
As a result, the Gradle retry plugin can not retrieve old failed test details by new rerun test details.

Issue:

 FAILURE: Build failed with an exception.
 * What went wrong:
 Execution failed for task ':functional-tests:functionalTest'.
  > The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues

As a solution, tests that have such kind of naming strategy should be modified.

Example:

...EnvExtension:-2 - Creating topologies (1)
   EnvExtension:-1 - Starting topology-related HC: lab_id=3
  EnvExtension:-1 - Topology-related HC passed: lab_id=3
...
BaseSpecification:-2 - Booked lab with id 3 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01
...
 INFO  LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]
 ERROR LogParallelSpecsListener:-2 - Specs ran in parallel: [HaFlowStatSpec]
  HaFlowStatSpec > System is able to collect latency stats for subflows ✘ System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0] (1m 1s)

    Condition failed with Exception:
    
    wait(statsRouterRequestInterval) { if(direction == FORWARD && subFlow.contains("haflow-a")) { assert labService.getLabs().first().labId != 3 } assert flowStats.of(subFlow).get(FLOW_RTT, direction).hasNonZeroValues() }
        at org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows(HaFlowStatSpec.groovy:100)
    Caused by: org.openkilda.functionaltests.helpers.Wrappers$WaitTimeoutException: Condition was not satisfied within 60.0 seconds
        at org.openkilda.functionaltests.helpers.Wrappers.wait(Wrappers.groovy:109)
        ... 1 more
    Caused by: Condition not satisfied:
    labService.getLabs().first().labId != 3 // intentional failure to trigger the 1st failure and during rerun the new topology is created and test should pass
    
    Retry by Gradle due to the failure: 
    HaFlowStatSpec STANDARD_OUT
 EnvExtension:-1 - Starting topology-related HC: lab_id=4
...
 BaseSpecification:-2 - Booked lab with id 4 for spec HaFlowStatSpec, thread: Thread[Test worker,5,main]. sw: 00:00:00:00:00:00:00:01

HaFlowStatSpec > System is able to collect latency stats for subflows > org.openkilda.functionaltests.spec.stats.HaFlowStatSpec.System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] STANDARD_OUT
    2024-04-11T09:20:22,280 INFO  LogParallelSpecsListener:-2 - Running test: HaFlowStatSpec['LOW_PRIORITY', 'HA_FLOW']#System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]
  HaFlowStatSpec > System is able to collect latency stats for subflows ✔ System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1] - //test passed after retry

> Task :functional-tests:functionalTest FAILED
:functional-tests:functionalTest (Thread[Execution worker for ':' Thread 4,5,main]) completed. Took 4 mins 20.883 secs.


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':functional-tests:functionalTest'.
> The following test methods could not be retried, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     org.openkilda.functionaltests.spec.stats.HaFlowStatSpec#System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]

Failed test:
System is able to collect latency stats for subflows [subFlow: 11Apr091705_157_oatflour2420_haflow-a, direction: FORWARD, #0]

Test name after Gradle rerun failed test^:
System is able to collect latency stats for subflows [subFlow: 11Apr091946_456_redlentils4861_haflow-b, direction: FORWARD, #1]

@yuliiamir
Copy link
Collaborator Author

The following method naming also caused the same Gradle issue with report generation in the scope of the retry mechanism.

@Unroll("Able to create a second flow if #data.description")
def "Able to create multiple flows on certain combinations of switch-port-vlans"() {

The appropriate changes have been added in the scope #5644

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant