Using soft assertions in TestNG

Posted: October 8th, 2009 | Author: Dave | Filed under: Guide | Tags: , , | 24 Comments »

One of the big differences between Selenium IDE and a Selenium RC solution is the ability to perform ‘soft’ assertions. Selenium IDE users can append commands with verify or assert to determine whether the test execution should stop when a failure is observed. A popular use for this is to first assert that you are on the correct page (assertTitle) and then verify elements on the page. If you were only able to assert then your tests may fail early on, not revealing further failures that may exist.

In Selenium RC you are limited by your test framework, and from my experience using the Java client library there isn’t a satisfactory equivalent to the soft assertion feature of Selenium IDE. Some solutions propose that you catch the assertions, log the occurrence of the failure, and check that there are no failures at the end of the test. The problem here is remembering to check for these verification failures.

Another solution suggests putting the check for verification failures in a method that is run by the test framework after every test, however when these fail (in TestNG) they are marked as configuration failures, and the default HTML report can still report your test suites as passed.

Cédric Beust (creator of TestNG) has discussed soft assertions on his blog, but most of the proposed solutions differ from the simple implementation that would encourage more Selenium IDE users to adopt Selenium RC.

TestNG has support for custom listeners, which can run when tests pass/fail/skip, as well as before and after invocation. By adding a custom listener to check for verification failures after invocation, we can get the details of all verification failures that have occurred, and report them at the same time as we report our hard failure, or if there are no hard failures we can change the result to a failure and report the verification failures.

This solution uses part of the TestNG soft failures patch by Dan Fabulich in order to combine the stack traces of multiple failures. Details of the patch are available here.

I have created a simple Eclipse project that can be downloaded, extracted and run. To keep it simple, this project does not use Selenium. You will need to add these files into your own project.

TestBase.java
This is the test base class, that imports TestNG and overrides the assert methods so that test classes extending this class are decoupled from TestNG. I have kept the methods in here to a minimum, in reality you will want to override more of the assert methods and have more equivalent verify methods.

package tests;
 
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
 
import org.testng.Assert;
import org.testng.ITestResult;
import org.testng.Reporter;
 
public class TestBase {
 
	private static Map> verificationFailuresMap = new HashMap>();
 
    public static void assertTrue(boolean condition) {
    	Assert.assertTrue(condition);
    }
 
    public static void assertFalse(boolean condition) {
    	Assert.assertFalse(condition);
    }
 
    public static void assertEquals(Object actual, Object expected) {
    	Assert.assertEquals(actual, expected);
    }
 
    public static void verifyTrue(boolean condition) {
    	try {
    		assertTrue(condition);
    	} catch(Throwable e) {
    		addVerificationFailure(e);
    	}
    }
 
    public static void verifyFalse(boolean condition) {
    	try {
    		assertFalse(condition);
	} catch(Throwable e) {
    		addVerificationFailure(e);
	}
    }
 
    public static void verifyEquals(Object actual, Object expected) {
    	try {
    		assertEquals(actual, expected);
		} catch(Throwable e) {
    		addVerificationFailure(e);
		}
    }
 
	public static List getVerificationFailures() {
		List verificationFailures = verificationFailuresMap.get(Reporter.getCurrentTestResult());
		return verificationFailures == null ? new ArrayList() : verificationFailures;
	}
 
	private static void addVerificationFailure(Throwable e) {
		List verificationFailures = getVerificationFailures();
		verificationFailuresMap.put(Reporter.getCurrentTestResult(), verificationFailures);
		verificationFailures.add(e);
	}
 
}

TestListenerAdapter.java
This adapter implements TestNG’s IInvokedMethodListener and overrides the two methods afterInvocation and beforeInvocation. This means that when you extend this class you only need to override the afterInvocation method.

package tests;
 
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
 
public class TestListenerAdapter implements IInvokedMethodListener {
 
	public void afterInvocation(IInvokedMethod arg0, ITestResult arg1) {}
 
	public void beforeInvocation(IInvokedMethod arg0, ITestResult arg1) {}
 
}

CustomTestListener.java
This is where the verification failures are combined for the report, and successful tests with verification failures are turned into failing tests.

package tests;
 
import java.util.List;
 
import org.testng.IInvokedMethod;
import org.testng.ITestResult;
import org.testng.Reporter;
import org.testng.internal.Utils;
 
public class CustomTestListener extends TestListenerAdapter {
 
	@Override
	public void afterInvocation(IInvokedMethod method, ITestResult result) {
 
		Reporter.setCurrentTestResult(result);
 
		if (method.isTestMethod()) {
 
			List verificationFailures = TestBase.getVerificationFailures();
 
			//if there are verification failures...
			if (verificationFailures.size() > 0) {
 
				//set the test to failed
				result.setStatus(ITestResult.FAILURE);
 
				//if there is an assertion failure add it to verificationFailures
				if (result.getThrowable() != null) {
					verificationFailures.add(result.getThrowable());
				}
 
				int size = verificationFailures.size();
				//if there's only one failure just set that
				if (size == 1) {
					result.setThrowable(verificationFailures.get(0));
				} else {
					//create a failure message with all failures and stack traces (except last failure)
					StringBuffer failureMessage = new StringBuffer("Multiple failures (").append(size).append("):nn");
					for (int i = 0; i < size-1; i++) {
						failureMessage.append("Failure ").append(i+1).append(" of ").append(size).append(":n");
						Throwable t = verificationFailures.get(i);
						String fullStackTrace = Utils.stackTrace(t, false)[1];
						failureMessage.append(fullStackTrace).append("nn");
					}
 
					//final failure
					Throwable last = verificationFailures.get(size-1);
					failureMessage.append("Failure ").append(size).append(" of ").append(size).append(":n");
					failureMessage.append(last.toString());
 
					//set merged throwable
					Throwable merged = new Throwable(failureMessage.toString());
					merged.setStackTrace(last.getStackTrace());
 
					result.setThrowable(merged);
				}
			}
		}
	}
 
}

VerifyTests.java
A few example tests that will all fail with verification or assertion failures. Run these tests to see the differences in the HTML report.

package tests;
 
import org.testng.annotations.Test;
 
public class VerifyTest extends TestBase {
 
	@Test
	public void test1() {
		verifyTrue(false);
		verifyEquals("pass", "fail");
		verifyFalse(true);
	}
 
	@Test
	public void test2() {
		verifyTrue(false);
		assertEquals("pass", "fail");
		verifyFalse(true);
	}
 
	@Test
	public void test3() {
		verifyTrue(true);
		verifyTrue(false);
		verifyTrue(true);
	}
 
	@Test
	public void test4() {
		assertTrue(true);
		assertTrue(false);
		assertTrue(true);
	}
 
}

Click here to download the above files, including dependencies, Eclipse project file, and Ant build.xml file.

Note
Unfortunately there is currently a bug in the TestNG Eclipse Plugin that means this method is not called, and the verification failures are ignored.

Update
The previously mentioned bug is now fixed and soft assertions should work as expected in Eclipse.

24 Comments on “Using soft assertions in TestNG”

  1. 1 Sergey Morozov said at 11:25 pm on January 8th, 2010:

    I was wondering if you managed to get it working with TestNG Eclipse Plugin. That bug seems to be marked as fixed, but I’m still having hard time to get it to work with latest versions of TestNG and TestNG Eclipse Plugin.

    Any info would be appreciated.

    Thank you.

  2. 2 Dave said at 5:21 pm on January 9th, 2010:

    Hi Sergey, thanks for leaving a comment. Yes, Cédric Beust fixed the bug and I can confirm that the latest version of the TestNG plugin will work with the soft assertions from this post.

    It would be worth checking that you have included the class in the preferences panel in Eclipse under custom listeners. It might also be worth adding a breakpoint in the method annotated as afterInvocation in CustomTestListener.java and run in debug.

    Hope that helps,

    Dave

  3. 3 Chinmayi said at 7:03 pm on April 30th, 2010:

    Hi,
    I have the same problem could not get this demo working..so I update testng plugin (5.12.06)
    I am using Eclipse 3.4.2 on Windows XP, i dont see the class under custom listeners under preferences panel.

    Thanks

  4. 4 Dave said at 8:00 pm on May 3rd, 2010:

    The latest Eclipse plugin uses a template XML file for the custom listeners. You can find the instructions for this here: http://testng.org/eclipse.html#eclipse-listeners

  5. 5 Mahesh Narayanan said at 8:41 pm on May 19th, 2010:

    Dave Thanks for this well thought out solution. I truly appreciate you for sharing this with us.

  6. 6 Pandaman said at 11:49 pm on July 21st, 2010:

    Thanks for the great share.

    Actually in the CustomTestListener.java, you need to comment out “import org.testng.TestListenerAdapter;” in order to make the @override work. (otherwise the “extends TestListenerAdapter” will think that you are extending “TestListenerAdapter” in “org.testng.TestListenerAdapter” ?!)
    I am using Netbeans tho.

  7. 7 Neil said at 4:59 pm on August 5th, 2010:

    Thanks for the great code … had fun playing with it this afternoon .. I had been put off using TestNG because of the soft assert issue but with your solution plan to investigate more what can be done with TestNG

  8. 8 Mikkel said at 1:21 pm on October 8th, 2010:

    Thanks alot for the code, was a great help to me :)
    I had quite a bit of problems with the custom listener thing, but finally figured out that i needed to add:

    to the testNG.xml file.

  9. 9 Hari said at 2:43 pm on March 1st, 2011:

    Hi Dave,

    First, thanks for this wonderful topic on soft assertion. Coming to my problem, I have been using selenium with java for around 3 to 4 months. So I am not well expert in both. I want a small help from you. How can I merge the coding you have given in this post with that of my recorded script given below. I am real need of this thing, as soon as possible.
    Btw, I am using Selenium RC with TestNG in Eclipse IDE.
    In this example, I am asserting “Google.co.in offered in: ” text in the google home page.

    Thanks,
    Hari

    Selenium code:

    package com.example.tests;

    import com.thoughtworks.selenium.*;
    import org.testng.annotations.*;
    import static org.testng.Assert.*;
    import java.util.regex.Pattern;

    public class google extends SeleneseTestNgHelper {
    @Test public void testGoogle() throws Exception {
    selenium.open(“/”);
    verifyTrue(selenium.isElementPresent(“hplogo”));
    verifyTrue(selenium.isElementPresent(“q”));
    verifyTrue(selenium.isElementPresent(“link=Advanced search”));
    verifyTrue(selenium.isElementPresent(“link=Language tools”));
    verifyTrue(selenium.isElementPresent(“btnG”));
    assertEquals(selenium.getText(“addlang”), “Google.co.in offered in: Hindi Bengali Telugu Marathi Tamil Gujarati Kannada Malayalam Punjabi”);
    assertEquals(selenium.getText(“//center[@id='fctr']/p”), “© 2011 – Privacy”);
    }
    }

  10. 10 eddard said at 6:56 pm on March 3rd, 2011:

    When should you really clear out the verificationFailuresMap?

  11. 11 Dave said at 1:34 pm on March 8th, 2011:

    Hi Hari, thanks for your comment.

    My example uses a TestBase.class, which the test class extends. In your code you are extending SeleneseTestNgHelper. To use soft assertions you will need to extend my TestBase instead. You will also need to specify a listener of CustomTestListener when running TestNG, see the official documentation for details.

  12. 12 Dave said at 1:40 pm on March 8th, 2011:

    Good question. Ive not actually used this code for some time but I don’t ever recall needing to clear out the map. What issue are you seeing? If you want to clear verificationFailuresMap then you could do it in a method marked with one of the @Before or @After annotations as documented here.

  13. 13 hemangi said at 1:50 pm on March 8th, 2011:

    Hi,
    Thanks for the Soft asserts.

    But when using this i am getting the error.

    Not sure how to fix this.

    Can you please help me out?

    java.lang.IllegalAccessError: tried to access method org.testng.Reporter.getCurrentTestResult()Lorg/testng/ITestResult; from class com.operativeone.Utilities.listener.TestBase
    at com.operativeone.Utilities.listener.TestBase.getVerificationFailures(TestBase.java:117)

  14. 14 Hari said at 12:09 pm on March 23rd, 2011:

    Hi Dave… Thanks for the reply and sorry for the delay in my reply… Your reference to the official documentation was helpful to me…
    Thank you once again…
    ~Harri…

  15. 15 Cedric said at 11:23 pm on March 28th, 2011:

    Thanks for the interesting article, I added a link to it in the TestNG documentation:

    http://testng.org/doc/misc.html

  16. 16 Dave said at 11:48 pm on March 28th, 2011:

    Thanks Cedric!

  17. 17 Ilya said at 11:45 am on April 13th, 2011:

    Anybody runs tests with this modification under ant? It seems listener fires on every method call in test method call queue. So it generates a lot of duplication text(logging record before test start for example) and exeptions list.

  18. 18 Dave said at 2:25 pm on April 13th, 2011:

    I ran this using Ant and didn’t have any unexpected issues, however I haven’t tried with the latest versions of TestNG and Ant.

  19. 19 P K said at 11:17 pm on April 25th, 2011:

    Thanks for sharing this code. Helped me a lot. However, I am getting the same issue as Ilya. For example, if my test method A failed with 2 exceptions, the report shows two instances of method A – one failed and one succeeded. Both having the same exception list.

    However, if method A succeeded, there is just one instance of A in the report – the succeeded one.

    Also, I am sure the test method is getting executed only once – it is only the report that has the duplicate exception list.

    Would you be able to help here. I am using the latest versions of Ant, TestNg and Java.

  20. 20 Dave said at 10:22 pm on May 10th, 2011:

    There might be an issue with the latest versions. I’ve not used this code myself for some time so I’m sorry but I can’t be much more use here. If you do find out the issue please comment with your findings and any solutions.

  21. 21 Bharath said at 3:54 pm on May 30th, 2011:

    Thank you for sharing the code.

  22. 22 Janusz said at 11:59 am on June 7th, 2011:

    Hi All,

    Thank you for sharing this code. Really useful stuff!

    I’ve got the same situation as P K.

    I’m using latest version of:
    TestNG, M2Eclipse, Java 1.6 64bit

    Anyone found solution for that?

    Cheers,
    J

  23. 23 Janusz said at 12:21 pm on June 7th, 2011:

    I solved the problem :)

    I forgot to add custom listener in testng.xml file :)

    You can also add parameter to your run configuration:
    -listener tests.CustomTestListener

    Cheers,
    J

  24. 24 bsc said at 2:06 pm on July 27th, 2011:

    Hi,
    Thx for your code, it’s really helpful.

    But I’m facing the same issue as P K
    I’m using Maven 2.1.1 and TestNG 6.1.1
    Issue is not reproducible when using TestNG 5.12.1

    Any solutions? Pls
    ————————-
    For example, if my test method A failed with 2 exceptions, the report shows two instances of method A – one failed and one succeeded. Both having the same exception list.
    However, if method A succeeded, there is just one instance of A in the report – the succeeded one.
    Also, I am sure the test method is getting executed only once – it is only the report that has the duplicate exception list.
    Would you be able to help here. I am using the latest versions of Ant, TestNg and Java.

Leave a Reply