From IDE to Selenium 2

Here are the slides from my ‘From IDE to Selenium 2′ talk that I gave at the second London Selenium Users meetup. Slides from the other talks to follow.

Full project containing Selenium IDE test script, Selenium 2 Java TestNG tests (including dependancies), and slides available in this public Subversion repository.

Another successful Selenium meetup in London!

Last night was the second London Selenium Users meetup. In contrast to the last one – a purely social event in a pub in Fitzrovia – this one was hosted by Google with talks from myself, David Burns (The Automated Tester), Eran Messeri (Google), and a Q&A with Simon Stewart (Google).

I arrived at Google’s London HQ (or Googleplex) at 18:00 to meet with Eran and Simon – who was only around thanks to the cloud of volcanic ash preventing him from flying out to New York! After a quick tour of the offices, and a brief label-off (I lost) people started to arrive. They were early – definitely a good sign!

I was the first to present, and I’m not used to speaking in-front of an audience, but my preparation and practise paid off as I feel my presentation went well. My talk covered a suggested migration path from Selenium IDE to Selenium 2 using my WebDriver Backed Formatters plugin for IDE, and contained live demos. After a few questions my presentation was over and I could relax!

David Burns next took to the stage with his presentation on what Selenium 2 means to the average QA. There was a short delay in getting started (which I understand we’re blaming Eran for, but might be more to do with David not using a Macbook ). Before long David’s talk was under way. Of particular interest were the demos, where by simply switching a test to use Selenium 2 he solved one of the most irritating limitations of Selenium 1 – typing into a rich text editor! This was followed by solving another limitation, with a demo that entered a file name into a <input type=”file”> field. After a few more questions for both David and Simon, the second presentation came to an end.

After our final break Eran gave a short presentation of his current work on an advanced user interactions API, which is something I’m looking forward to for testing complex Ajax applications with rich interfaces. The idea being that you can combine several actions such as ‘Hold shift, click down on element X, and drag it onto element Y’. Also mentioned was the mobile (or ‘Fingers’) interactions API to support swiping, pinching, etc. A very good talk, and proof (if needed) that Google are taking the future of Selenium very seriously – there’s a lot of work going into it!

Earlier on in the evening Simon had asked for questions to be written on the whiteboard, and instead of his planned talk – which he’d mostly given all of the details out as answers to questions throughout the evening – he worked his way through the list. The questions included: testing Ajax applications using Selenium 2 (you can), is the Selenium IDE dead (no), is Selenese dead (not quite), and does Selenium Grid support Selenium 2 (not yet).

After the Q&A it there was just time for some more refreshments, but then I had to then start my journey home. I hear those that stayed had some lively discussions in the nearby pub about dynamic Firefox instances and the like, with Google buying everyone one more beer!

Feedback on the event has so far been very positive, and I’d like to thank everyone that came along – there were some interesting questions, and a good range of Selenium experience in the room.

It was a thoroughly enjoyable event and I’d like to thank Google for being such awesome hosts, Simon for all of his efforts and question answering, Eran Messeri for his great work on the project and for sharing some of that with us, and David Burns for his presentation and demos.

I would also like to thank Jérémy Hérault (Selenium committer) and Kévin Pollet (creator of Helium plugin for Selenium IDE) for coming along – they were travelling for 8 hours to get to the event and had to do a return journey the same day! I hope it hasn’t put you off coming to future events guys!

Slides, notes, demos, videos (possibly), all to follow!

Selenium IDE – My plugin baby

With last month’s release of Selenium IDE 1.0.4 came initial support for plugins, and this month’s release of 1.0.5 continues to build on the new plugin API as well as fixing a few bugs.

This is quite possibly the most exciting development in Selenium IDE in a long time, and the best thing is that it enables users to come up with their own improvements – in much the same way as Greasemonkey or Firefox addons themselves. The possibilities are endless.

Adam Goucher continues to post guidance on creating plugins on his blog as the API evolves. I have followed these myself to create two plugins:

• Flow control – This is actually just an old Selenium user extension that I’ve repackaged into a plugin. Two main advantages of doing this are: an easy install/disable/uninstall process, and no messing around with JavaScript files! [install] [public repository]
• WebDriver Backed Selenium Formatters – This is a formatter plugin that provides the Selenium emulation from WebDriver, giving quick access to WebDriver’s advantages without changing from the existing Selenium API. [install] [public repository]

These plugins are available to download from SeleniumExamples, and you can view the source/report issues/provide feedback via the GitHub repositories.

Let SeleniumExamples know if you’re developing your own plugin, or have any good ideas. I’ll be happy to review, host, or link to your plugins – most of all we want to encourage plugin development.

Update:
Plugins now made available via addons.mozilla.org:

• Flow Control
• WebDriver Backed Selenium Formatters

New London Users Event: Introducing Selenium 2

Today the ‘Introducing Selenium 2′ London Selenium Users event was tentatively announced for April 20th 2010. Please visit the event page to register your interest. The date will be confirmed when a venue has been decided. If you would like to suggest a venue, please send me a message.

Selenium 2 Examples

Yesterday Selenium 2 (alpha 1) was released. This is the first release since the Selenium and WebDriver projects started to merge. The main difference is the inclusion of the WebDriver API into Selenium. I’ve put together a small example below that uses the new API to log into two web based e-mail clients and send an e-mail.

Read the rest of this entry »

Improving Selenium Grid

Hopefully Selenium Grid 1.0.5 will soon be released, with the much anticipated self-healing features that will save me so much time when RCs go AWOL. Looking further ahead, I would like to see some minor improvements to the Selenium Grid Console such as integrating this very handy Greasemonkey script for unregistering Remote Controls and sortable columns. Below is a quick mock-up of how the console could look with these simple changes.

The above changes I’m pretty confident I could make myself, but after trying to get Selenium Grid with a resource handler for a couple of hours I decided instead I’d mock it up and make a public plea for these changes!

While I’m at it, something I’m less certain I’d be able to do myself is add the queued requests to the console with the ability to delete them. It’s not uncommon for someone to fire off a test suite and when the first few tests appear to be failing, cancel the process. Unfortunately this means that several RCs are unable to complete, and Grid still has pending requests. In the past I’ve started X number of RCs in order to consume the remaining requests and then kill them, however I usually just restart the Grid and all RCs. Below is an idea of how this could look in the console.

A Successful First London Selenium Meetup

The first London Selenium meetup took place on the 11th November. If you weren’t able to make it then due to the success I will be scheduling another one early in the new year.

From the feedback I’ve received it seems people would like some sort of presentation or demo at the next event. If you would like to volunteer to talk at the next event please get in touch via the Meetup.com group page. Please also get in touch if you have suggestions for a venue, or suggestions for topics that you’d like to hear about.

Some suggested topics are below. Soon I will combine these with any other suggestions into a poll to see what is most popular. You can make further suggestions by responding to this message.

1. Stop pausing – testing AJAX with Selenium by waiting instead of pausing
2. Minimising impact of UI changes using the UI-Elements map
3. Introduction to locating elements by XPath and CSS
4. Integrating Selenium tests into a continuous build
5. Battle of the testing frameworks – what’s available and how to choose
6. Selenium in the Grid – How to get Selenium Grid working, and how to test in the Cloud
7. Selenium vs QTP – What are the main differences? The pros and cons of each
8. Selenium 2.0 examples – Demonstration of what’s available in WebDriver/Selenium 2.0
9. Introducing Selenium to an organization. Challenges, steps, selling Selenium to management
10. Testing best practices
11. How to balance effort required to create tests with benefits

You can see a few photos of the first meetup here. I have also set up a poll asking Should we have name labels for future events?

Finally, I’d like to once again thank the sponsors Paul Humphreys from Clear Cube Consulting & Phil Smith from Cloud Testing for their support by providing venue and refreshments costs for the first event. I hope they benefited from the event as much as everyone else!

Construct unique safe filenames in TestNG

In order to save files for investigating failures from within TestNG it’s important to have a safe filename that is unique to the test – otherwise you may overwrite important files. I have written the following simple method in Java that is called from a listener with an ITestResult parameter to construct a unique file name that should be safe on most file systems.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

private static String fileNameFrom(ITestResult result, String fileExtension) {
  List<String> parts = new ArrayList<String>();
  parts.add(result.getTestClass().getName());
  parts.add(result.getMethod().getMethodName());
 
  //add parameters
  Object[] parameters = result.getParameters();
  for (Object parameter: parameters)
    parts.add(parameter != null ? parameter.toString() : NULL_PARAMETER_VALUE);
 
  String fileName = StringUtils.join(parts.toArray(new String[parts.size()]), ".");
  fileName = fileName.replaceAll("[\\^\\\\.\\-:;#_]", "_");
  return fileName + "." + fileExtension;
}

Capturing screenshots from remote Selenium RC

Despite the name, the Selenium RC (Remote Control) server is often run on the same machine as the testing framework, which makes saving screenshots to disk quite easy. If however you are running Selenium RC on a separate machine, or are using Selenium Grid it can become more difficult as the screenshots are also saved on the remote machines.

To solve this you can use the captureScreenshotToString and captureEntirePageScreenshotToString commands, which return a Base64 encoded String of the screenshot, which you can then decode and save to disk on your testrunner machine.

The following demonstrates the latter command (entire screenshot) in Java. I have added this to a TestNG afterInvocation listener for failed tests so that I have a screenshot of the page that resulted in the failure, which can be very valuable for diagnosing issues.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public void afterInvocation(IInvokedMethod method, ITestResult result) {
  if (!result.isSuccess()) {
    String imageName = "screenshot.png";
    String imagePath = outputDirectory + separator + imageName;
    try {
      String base64Screenshot = session().captureEntirePageScreenshotToString("");
      byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
      FileOutputStream fos = new FileOutputStream(new File(imagePath));
      fos.write(decodedScreenshot);
      fos.close();
      Reporter.log("<a href=\"file:///" + imagePath + "\">Screenshot</a>");
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

Notes:

• This example uses org.apache.commons.codec.binary.Base64
• In this example, outputDirectory is set in a onStart listener from the ITestContext parameter’s getOutputDirectory method
• In reality you’d want to construct imageName from the test method name so that it is unique for each test
• The captureEntireScreenshot* commands have limited browser support. Currently I only capture these screenshots if the browser is Firefox

Using soft assertions in TestNG

One of the big differences between Selenium IDE and a Selenium RC solution is the ability to perform ‘soft’ assertions. Selenium IDE users can append commands with verify or assert to determine whether the test execution should stop when a failure is observed. A popular use for this is to first assert that you are on the correct page (assertTitle) and then verify elements on the page. If you were only able to assert then your tests may fail early on, not revealing further failures that may exist.

In Selenium RC you are limited by your test framework, and from my experience using the Java client library there isn’t a satisfactory equivalent to the soft assertion feature of Selenium IDE. Some solutions propose that you catch the assertions, log the occurrence of the failure, and check that there are no failures at the end of the test. The problem here is remembering to check for these verification failures.

Another solution suggests putting the check for verification failures in a method that is run by the test framework after every test, however when these fail (in TestNG) they are marked as configuration failures, and the default HTML report can still report your test suites as passed.

Cédric Beust (creator of TestNG) has discussed soft assertions on his blog, but most of the proposed solutions differ from the simple implementation that would encourage more Selenium IDE users to adopt Selenium RC.

TestNG has support for custom listeners, which can run when tests pass/fail/skip, as well as before and after invocation. By adding a custom listener to check for verification failures after invocation, we can get the details of all verification failures that have occurred, and report them at the same time as we report our hard failure, or if there are no hard failures we can change the result to a failure and report the verification failures.

This solution uses part of the TestNG soft failures patch by Dan Fabulich in order to combine the stack traces of multiple failures. Details of the patch are available here.

I have created a simple Eclipse project that can be downloaded, extracted and run. To keep it simple, this project does not use Selenium. You will need to add these files into your own project.

Read the rest of this entry »