Using soft assertions in TestNG

One of the big differences between Selenium IDE and a Selenium RC solution is the ability to perform ‘soft’ assertions. Selenium IDE users can append commands with verify or assert to determine whether the test execution should stop when a failure is observed. A popular use for this is to first assert that you are on the correct page (assertTitle) and then verify elements on the page. If you were only able to assert then your tests may fail early on, not revealing further failures that may exist.

In Selenium RC you are limited by your test framework, and from my experience using the Java client library there isn’t a satisfactory equivalent to the soft assertion feature of Selenium IDE. Some solutions propose that you catch the assertions, log the occurrence of the failure, and check that there are no failures at the end of the test. The problem here is remembering to check for these verification failures.

Another solution suggests putting the check for verification failures in a method that is run by the test framework after every test, however when these fail (in TestNG) they are marked as configuration failures, and the default HTML report can still report your test suites as passed.

Cédric Beust (creator of TestNG) has discussed soft assertions on his blog, but most of the proposed solutions differ from the simple implementation that would encourage more Selenium IDE users to adopt Selenium RC.

TestNG has support for custom listeners, which can run when tests pass/fail/skip, as well as before and after invocation. By adding a custom listener to check for verification failures after invocation, we can get the details of all verification failures that have occurred, and report them at the same time as we report our hard failure, or if there are no hard failures we can change the result to a failure and report the verification failures.

This solution uses part of the TestNG soft failures patch by Dan Fabulich in order to combine the stack traces of multiple failures. Details of the patch are available here.

I have created a simple Eclipse project that can be downloaded, extracted and run. To keep it simple, this project does not use Selenium. You will need to add these files into your own project.

Read the rest of this entry »