In one of our Java project we use TestNG's dataprovider and retry mechanism quite a lot. We came accross a weird behaviour and can't decide whether it is a bug, or expected behaviour of TestNG.
In short: When running a test with dataprovider and retries, if a test for a certain given data-provider value eventually passes after a retry, it still gets generated in the testng-failed.xml if that method has a failed invocation (after all retries) with a different value.
Let me share the code I have.
RetryAnalyzer.java
import org.testng.IRetryAnalyzer;import org.testng.ITestResult;public class RetryAnalyzer implements IRetryAnalyzer { int counter = 0; int retryLimit = 1; /* * Retry method */ public boolean retry(ITestResult result) { if (!result.isSuccess()) { if (counter < retryLimit) { System.out.println(" ... FAILURE! Retrying ... (#" + (counter + 1) +" out of " + retryLimit +")"); counter++; return true; } } return false; }}RetryTest.java
import static org.testng.Assert.assertEquals;import java.util.Random;import org.testng.annotations.DataProvider;import org.testng.annotations.Test;public class RetryTest { @Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dp") public void testDataProvider(int number) { Random random = new Random(); int randomNumber = random.nextInt(3) + 1; // Generates a number between 1 and 3 System.out.println("Number from data-provider: " + number); System.out.println("Generated number: " + randomNumber); assertEquals(randomNumber, number, "The two values are different!"); } @DataProvider(name = "dp") public Object[][] createTestData() { return new Object[][] { { 1 }, { 2 }, { 3 } }; }}We should examine a scenario when one of the test values completely fails (even after a retry), but the other one (i.e.: invocation with another data-provider value) passes after a failure.
Number from data-provider: 1Generated number: 2 ... FAILURE! Retrying ... (#1 out of 1)Number from data-provider: 1Generated number: 1Number from data-provider: 2Generated number: 1 ... FAILURE! Retrying ... (#1 out of 1)Number from data-provider: 2Generated number: 1Number from data-provider: 3Generated number: 1 ... FAILURE! Retrying ... (#1 out of 1)Number from data-provider: 3Generated number: 3PASSED: restAPITest.boards.BoardsTest.testDataProvider(3)PASSED: restAPITest.boards.BoardsTest.testDataProvider(1)FAILED: restAPITest.boards.BoardsTest.testDataProvider(2)As it can be seen, for data-provider value '2', both test-runs failed, since overall it FAILED. But for the values '1' and '3' the test eventually succeded after a failure.But when I open the testng-failed.xml file which is generated after this test run, I can see the following:
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd"><suite name="Failed suite [Default suite]" guice-stage="DEVELOPMENT"><test thread-count="5" name="Default test(failed)"><classes><class name="RetryTest"><methods><include name="testDataProvider" invocation-numbers="0 1 2"/></methods></class> <!-- RetryTest --></classes></test> <!-- Default test(failed) --></suite> <!-- Failed suite [Default suite] -->Notice the invocation-numbers part which includes all the indexes (i.e.: the 0th, 1st, and 2nd value of the data-provider) to be executed for a rerun.I would expect that invocation-numbers would contain only those indexes that failed completely - meaning after all retries. (In this case only the index value '1' should be there, since that represents the value '2' of the data-provider.)
Can somebody clarify me this behaviour?