Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

onTestFailure not being called when test is retried (6.9.6) #857

Closed
jineshqa opened this issue Nov 5, 2015 · 23 comments · May be fixed by #860
Closed

onTestFailure not being called when test is retried (6.9.6) #857

jineshqa opened this issue Nov 5, 2015 · 23 comments · May be fixed by #860

Comments

@jineshqa
Copy link

jineshqa commented Nov 5, 2015

My Test :
@listeners({ ScreenshotListener.class })
public class TestRetryLogic {
@test(retryAnalyzer = Retry.class)
public void testRetryLogic() {
fail("Failing");
}
}

My Retry Analyzer :
public class Retry implements IRetryAnalyzer {
private final int maxRetryCountPerTest = 1;
private int retryCountPerTest = 0;

public boolean retry(ITestResult result) {
    if(retryCountPerTest < maxRetryCountPerTest) {
        retryCountPerTest++;
        return true;
    }
    return false;
}

}

My Listener :
public class ScreenshotListener extends TestListenerAdapter {

@Override
public void onTestFailure(ITestResult tr) {
    System.out.println("Test FAILED");
}

@Override
public void onTestSkipped(ITestResult tr) {
    System.out.println("Test SKIPPED");
}

@Override
public void onTestStart(ITestResult result) {
    System.out.println("Test STARTED");
}

}

My output:
Test STARTED
Test SKIPPED
Test STARTED
SKIPPED: testRetryLogic
java.lang.AssertionError: Failing

I would have expected :

Test STARTED
Test SKIPPED
Test STARTED
Test FAILED (Missing from above)
SKIPPED: testRetryLogic
java.lang.AssertionError: Failing

Is this a bug? Or is this expected?

Thank you.

@juherr
Copy link
Member

juherr commented Nov 5, 2015

If the behavior was different before, it is a bug.

@jineshqa
Copy link
Author

jineshqa commented Nov 5, 2015

It certainly was. I have code to take screenshot and add that using reporter.log in testng results. Now, none of that is working :-(

@juherr
Copy link
Member

juherr commented Nov 5, 2015

Ok, thank. I will have a look asap.

juherr pushed a commit to juherr/testng that referenced this issue Nov 5, 2015
@juherr
Copy link
Member

juherr commented Nov 5, 2015

@jineshkenandy I tried to reproduce your issue without success. See #860 and tell me if you see something bad.

@juherr juherr added the 3 - Done label Nov 5, 2015
@jineshqa
Copy link
Author

jineshqa commented Nov 5, 2015

@juherr Could you put a print statement inside your onTestFailure method? Count has never been the issue it counts correctly for some reason.

Another thing I observed is that you are calling testng from code and not xml. All of us who faced the issue are calling it from XML.

Thanks,
Jinesh

@juherr
Copy link
Member

juherr commented Nov 5, 2015

No need to print: the method is called and stored.
But you can try on your own side by pulling my branch.

+1 about xml, I will try. It should not change anything but I've already seen xml only problems.

@jineshqa
Copy link
Author

jineshqa commented Nov 5, 2015

@juherr
I set my retry analyzer using below code. I noticed that when I set retry analyzer print statement inside onTestFailure is not executed. However if retry analyzer is not set print statement is executed correctly.

@BeforeTest(alwaysRun = true)
public void beforeTest(ITestContext context) {
    for (ITestNGMethod method : context.getAllTestMethods()) {
        method.setRetryAnalyzer(new utils.common.RetryAnalyzer());
    }
}

@juherr
Copy link
Member

juherr commented Nov 6, 2015

It is really an uncommon way the initialize it? Can I ask why do you do it?
IMO, you should use annotation transformer instead.

BTW, about the print, can we suppose it is a display issue?

However if retry analyzer is not set print statement is executed correctly

I don't understand what you want to say.

@juherr
Copy link
Member

juherr commented Nov 6, 2015

@jineshkenandy I updated the sample with what I understood of your comments, but I confirm the result is still the same for me:

START
SKIPPED
START
FAILURE

@szaluk
Copy link

szaluk commented Nov 6, 2015

I am initializing my Retry Analyzer the same way as @jineshkenandy I am seeing the same thing. In a previous version of TestNG I handled the retries in the onTestFailed() method but when I updated to version 6.9.6 I noticed that wasn't getting called anymore. I have since had to add code to the onTestSkipped() and the onTestFailedButWithinSuccessPercentage() to handle my retries. It's working fine now but I was surprised by the change.

@juherr
Copy link
Member

juherr commented Nov 6, 2015

@jineshkenandy @szaluk Could you try the latest testng version (6.9.9) and tell me if you still observe the same problem?

Because, as proved by #860, the problem doesn't occur on the master branch.

@szaluk
Copy link

szaluk commented Nov 6, 2015

It does seem to be fixed in 6.9.9. I created a very simple project to test it out that does 2 retries.

This is what I am seeing in 6.9.6:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailedButWithinSuccessPercentage()

This is what I am seeing in 6.9.9:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailure()

I will upgrade to 6.9.9. Thanks!

--Steve

BTW, I looked at the change log for TestNG and there is nothing in there for the releases > 6.9.6 that mentions a fix for retried tests: https://github.com/cbeust/testng/blob/master/CHANGES.txt

Am I looking in the wrong spot?

@jineshqa
Copy link
Author

jineshqa commented Nov 6, 2015

@juherr I meant that if I do not use the retry analyzer at all and let test just fail. The print statement inside onTestFailure gets executed.

@szaluk Thanks for pitching in. I am having exact same issue. I'll upgrade and get back with my findings.

@jineshqa
Copy link
Author

jineshqa commented Nov 7, 2015

@juherr Same result with 6.9.9 as well and I moved to annotation transformer.

@jineshqa
Copy link
Author

jineshqa commented Nov 7, 2015

@szaluk Could you confirm that 6.9.9 is working fine for you on actual project and not test project?

@szaluk
Copy link

szaluk commented Nov 7, 2015

Sure. I will be able to try that tomorrow. I will let you know what I see.

Thanks,
Steve

@szaluk
Copy link

szaluk commented Nov 9, 2015

@jineshkenandy - It does indeed seem to be working for me with 6.9.9 in my main project. This is how I am setting up my retry anaylzer:

@BeforeClass(alwaysRun = true)
public void baseTestClassSetup(ITestContext testContext) {
    for(ITestNGMethod testNGMethod : testContext.getAllTestMethods()) {
        if(testNGMethod.getRetryAnalyzer() == null)
            testNGMethod.setRetryAnalyzer(new RetryAnalyzer());
    }
}

This is how you were originally doing it before you changed over to annotation transformer.

@jineshqa
Copy link
Author

jineshqa commented Nov 9, 2015

@szaluk Are you using testng to run selenium webdriver?

@szaluk
Copy link

szaluk commented Nov 9, 2015

@jineshkenandy - Yes. It's a Maven based project using Java 7 and Selenium WebDriver 2.48.2

@jineshqa
Copy link
Author

jineshqa commented Nov 9, 2015

@szaluk I am using Selenium WebDriver 2.48.2 with testNg, no luck whatsoever.

@juherr
Copy link
Member

juherr commented Nov 10, 2015

Guys, should we understand the issue is fixed with 6.9.9?

@jineshqa
Copy link
Author

@juherr Please mark it as closed. Thanks for all the help.

@ghost
Copy link

ghost commented Dec 4, 2015

Hi,

I can reproduce it in 6.9.9 as well.

Best regards,

Baubak

From: Steven Zaluk <[email protected]mailto:[email protected]>
Reply-To: cbeust/testng <[email protected]mailto:[email protected]>
Date: Friday 6 November 2015 at 18:51
To: cbeust/testng <[email protected]mailto:[email protected]>
Subject: Re: [testng] onTestFailure not being called when test is retried (6.9.6) (#857)

It does seem to be fixed in 6.9.9. I created a very simple project to test it out that does 2 retries.

This is what I am seeing in 6.9.6:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailedButWithinSuccessPercentage()

This is what I am seeing in 6.9.9:
Inside onTestSkipped()
Inside onTestSkipped()
Inside onTestFailure()

I will upgrade to 6.9.9. Thanks!

--Steve

BTW, I looked at the change log for TestNG and there is nothing in there for the releases > 6.9.6 that mentions a fix for retried tests: https://github.com/cbeust/testng/blob/master/CHANGES.txt

Am I looking in the wrong spot?


Reply to this email directly or view it on GitHubhttps://github.com//issues/857#issuecomment-154484237.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants