Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testng-failed.xml does not contain failed methods from all the classes while running the tests in parallel. #2230

Closed
1 of 7 tasks
deepakguna opened this issue Jan 22, 2020 · 4 comments
Closed
1 of 7 tasks

testng-failed.xml does not contain failed methods from all the classes while running the tests in parallel. #2230

deepakguna opened this issue Jan 22, 2020 · 4 comments
Labels
Need: sample
Milestone
7.2

Comments

@deepakguna
Copy link

deepakguna commented Jan 22, 2020 •
edited by krmahadevan
Loading

TestNG Version

7.1.0

Eclipse version

Version: 2019-06 (4.12.0)
Build id: 20190614-1200

TestNG for Eclipse Plugin

Version: TestNG 7.0.0.201908240652

Expected behavior

testng-failed.xml should contain failed methods from all the classes while running the tests in parallel.

JDK 13.0.1 Testng 7.1.0

When running the tests with Testng there are failures from the methods in both Class A & B, how ever testng-failed.xml contains only either of the class? What may go wrong here

Actual behavior

testng-failed.xml is inconsistent while running in parallel. On Many occasion it includes failures from only one test/class while there are failures in other test/class.

Is the issue reproductible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans

Test case sample

I have two test classes TestRed and TestRed . Each of the class have 2 @test methods and one of them will fail. testng-failed.xml is inconsistent while running in parallel. On Many occasion it includes failures from only one test/class while there are failures in other test/class.

package test;

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestRed {
	
	@Test
	public void testAInRed() {
		Assert.assertTrue(false, "will fail");
	}
	@Test
	public void testBInRed() {
		Assert.assertTrue(true, "will pass");
	}
}

Here is my suite xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Mini Regression" parallel="tests" thread-count="3" >
<!-- parallel="tests" thread-count="3" -->
	<test name="Test Blue">
		<classes>
			<class
				name="test.TestBlue" />
		</classes>
	</test>
	<test name="Test Red">
		<classes>
			<class
				name="test.TestRed" />
		</classes>
	</test> 	
</suite>

Below is the testng-failed.xml :

<suite thread-count="3" parallel="tests" name="Failed suite [Mini Regression]" guice-stage="DEVELOPMENT">
  <test thread-count="3" parallel="tests" name="Test Blue(failed)">
    <classes>
      <class name="test.TestBlue">
        <methods>
          <include name="testAInBlue"/>
        </methods>
      </class> <!-- test.TestBlue -->
    </classes>
  </test> <!-- Test Blue(failed) -->
</suite> <!-- Failed suite [Mini Regression] -->

It works fine in 6.14.3 but not in 7.1.0
@krmahadevan
Copy link
Member

I have not been able to recreate this issue using 7.1.0

Please feel free to edit this sample to turn it into a reproducible sample. If that cant be done, then am afraid we will have to close this issue.

Here's what I used:

Test classes

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestBlue {
  @Test
  public void testAInBlue() {
    Assert.fail("will fail");
  }

  @Test
  public void testBInBlue() {
    Assert.assertTrue(true, "will pass");
  }
}
import org.testng.Assert;
import org.testng.annotations.Test;

public class TestRed {
  @Test
  public void testAInRed() {
    Assert.fail("will fail");
  }

  @Test
  public void testBInRed() {
    Assert.assertTrue(true, "will pass");
  }
}

Test Runner to run the test

import java.io.File;
import java.io.IOException;
import java.util.Collections;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import static org.assertj.core.api.Assertions.assertThat;

import org.assertj.core.util.Files;
import org.testng.TestNG;
import org.testng.annotations.Test;
import org.testng.xml.XmlClass;
import org.testng.xml.XmlSuite;
import org.testng.xml.XmlSuite.ParallelMode;
import org.testng.xml.XmlTest;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;

public class IssueTest {
  @Test
  public void runTest()
      throws ParserConfigurationException, SAXException, IOException, XPathExpressionException {
    String version =
        new File(TestNG.class.getProtectionDomain().getCodeSource().getLocation().getFile())
            .getParentFile()
            .getName();
    assertThat(version).isEqualTo("7.1.0");
    TestNG testng = new TestNG();
    testng.setVerbose(2);
    XmlSuite xmlSuite = suite();
    test(xmlSuite, "Test Blue", TestBlue.class);
    test(xmlSuite, "Test Red", TestRed.class);
    System.err.println("The suite xml file will look like below");
    System.err.println(xmlSuite.toXml());
    String outputDir = Files.newTemporaryFolder().getAbsolutePath();
    testng.setXmlSuites(Collections.singletonList(xmlSuite));
    testng.setOutputDirectory(outputDir);
    testng.setUseDefaultListeners(true);
    testng.run();
    Document doc = asDocument(outputDir);
    String m1 =
        extractFailedMethodName(
            doc, "//test[@name='Test Red(failed)']/classes/class/methods/include/@name");
    assertThat(m1).isEqualTo("testAInRed");
    String m2 =
        extractFailedMethodName(
            doc, "//test[@name='Test Blue(failed)']/classes/class/methods/include/@name");
    assertThat(m2).isEqualTo("testAInBlue");
  }

  private static Document asDocument(String outputDir)
      throws IOException, ParserConfigurationException, SAXException {
    File failedXml = new File(outputDir + File.separator + "testng-failed.xml");
    System.err.println("Printing the contents of [testng-failed.xml]");
    java.nio.file.Files.readAllLines(failedXml.toPath()).forEach(System.err::println);
    DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
    DocumentBuilder builder = factory.newDocumentBuilder();
    return builder.parse(failedXml);
  }

  private static String extractFailedMethodName(Document doc, String expression)
      throws XPathExpressionException {
    XPath xPath = XPathFactory.newInstance().newXPath();
    return (String) xPath.compile(expression).evaluate(doc, XPathConstants.STRING);
  }

  private static void test(XmlSuite xmlSuite, String name, Class<?> testClass) {
    XmlTest xmlTest = new XmlTest(xmlSuite);
    xmlTest.setName(name);
    xmlTest.setXmlClasses(Collections.singletonList(new XmlClass(testClass.getName())));
  }

  private static XmlSuite suite() {
    XmlSuite xmlSuite = new XmlSuite();
    xmlSuite.setName("Mini Regression");
    xmlSuite.setParallel(ParallelMode.TESTS);
    xmlSuite.setThreadCount(3);
    return xmlSuite;
  }
}

Execution output:

The suite xml file will look like below
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite thread-count="3" name="Mini Regression" parallel="tests">
  <test thread-count="3" name="Test Blue" parallel="tests">
    <classes>
      <class name="com.rationaleemotions.github.issue2230.TestBlue"/>
    </classes>
  </test> <!-- Test Blue -->
  <test thread-count="3" name="Test Red" parallel="tests">
    <classes>
      <class name="com.rationaleemotions.github.issue2230.TestRed"/>
    </classes>
  </test> <!-- Test Red -->
</suite> <!-- Mini Regression -->

PASSED: testBInBlue
FAILED: testAInBlue
java.lang.AssertionError: will fail
	at org.testng.Assert.fail(Assert.java:97)
	at com.rationaleemotions.github.issue2230.TestBlue.testAInBlue(TestBlue.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:816)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:766)
	at org.testng.TestRunner.run(TestRunner.java:587)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:28)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:425)
	at org.testng.internal.thread.ThreadUtil.lambda$execute$0(ThreadUtil.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

PASSED: testBInBlue
PASSED: testBInRed
FAILED: testAInRed
java.lang.AssertionError: will fail
	at org.testng.Assert.fail(Assert.java:97)
	at com.rationaleemotions.github.issue2230.TestRed.testAInRed(TestRed.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:816)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:766)
	at org.testng.TestRunner.run(TestRunner.java:587)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:28)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:425)
	at org.testng.internal.thread.ThreadUtil.lambda$execute$0(ThreadUtil.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

FAILED: testAInBlue
java.lang.AssertionError: will fail
	at org.testng.Assert.fail(Assert.java:97)
	at com.rationaleemotions.github.issue2230.TestBlue.testAInBlue(TestBlue.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:816)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:766)
	at org.testng.TestRunner.run(TestRunner.java:587)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:28)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:425)
	at org.testng.internal.thread.ThreadUtil.lambda$execute$0(ThreadUtil.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


===============================================
    Test Blue
    Tests run: 2, Failures: 1, Skips: 0
===============================================


===============================================
    Test Red
    Tests run: 4, Failures: 2, Skips: 0
===============================================

PASSED: testBInBlue
PASSED: testBInRed
FAILED: testAInRed
java.lang.AssertionError: will fail
	at org.testng.Assert.fail(Assert.java:97)
	at com.rationaleemotions.github.issue2230.TestRed.testAInRed(TestRed.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:816)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:766)
	at org.testng.TestRunner.run(TestRunner.java:587)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:28)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:425)
	at org.testng.internal.thread.ThreadUtil.lambda$execute$0(ThreadUtil.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

FAILED: testAInBlue
java.lang.AssertionError: will fail
	at org.testng.Assert.fail(Assert.java:97)
	at com.rationaleemotions.github.issue2230.TestBlue.testAInBlue(TestBlue.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:134)
	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:597)
	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:173)
	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:816)
	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:146)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at org.testng.TestRunner.privateRun(TestRunner.java:766)
	at org.testng.TestRunner.run(TestRunner.java:587)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
	at org.testng.SuiteRunner.access$000(SuiteRunner.java:28)
	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:425)
	at org.testng.internal.thread.ThreadUtil.lambda$execute$0(ThreadUtil.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


===============================================
    Test Red
    Tests run: 4, Failures: 2, Skips: 0
===============================================


===============================================
Mini Regression
Total tests run: 4, Passes: 2, Failures: 2, Skips: 0
===============================================

Printing the contents of [testng-failed.xml]
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite thread-count="3" verbose="2" name="Failed suite [Mini Regression]" parallel="tests">
  <test thread-count="3" verbose="2" name="Test Red(failed)" parallel="tests">
    <classes>
      <class name="com.rationaleemotions.github.issue2230.TestRed">
        <methods>
          <include name="testAInRed"/>
        </methods>
      </class> <!-- com.rationaleemotions.github.issue2230.TestRed -->
    </classes>
  </test> <!-- Test Red(failed) -->
  <test thread-count="3" verbose="2" name="Test Blue(failed)" parallel="tests">
    <classes>
      <class name="com.rationaleemotions.github.issue2230.TestBlue">
        <methods>
          <include name="testAInBlue"/>
        </methods>
      </class> <!-- com.rationaleemotions.github.issue2230.TestBlue -->
    </classes>
  </test> <!-- Test Blue(failed) -->
</suite> <!-- Failed suite [Mini Regression] -->


===============================================
Default Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================


Process finished with exit code 0

@krmahadevan krmahadevan added this to the 7.2 milestone Jan 27, 2020
@deepakguna
Copy link
Author

@krmahadevan Thank you I was not able to reproduce this issue with the test runner. However i was consistently able to reproduce this issue 2/5 times by running from the suite.xml [it triggers via eclipse-testng plugin] ?

Should this go under : https://github.com/cbeust/testng-eclipse/issues ?

@krmahadevan
Copy link
Member

krmahadevan commented Jan 27, 2020 •
edited
Loading

@deepakguna - Yes. It may have to be moved to the eclipse plugin project if this is happening consistently with the plugin. Please feel free to close this issue and create a new one in the testng-eclipse project.

@deepakguna
Copy link
Author

@krmahadevan Thank you! I have moved the issue to testng_eclipse plugin - testng-team/testng-eclipse#473

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Need: sample
Projects
None yet
Milestone
7.2
Development

No branches or pull requests
2 participants
@krmahadevan @deepakguna