ReportNG Enrichment With Screenshots

Having a vast number of regression tests it became obvious to our QA team that a good reporting tool for reporting the execution results was of great essence. Our first approach in reporting was to use a simple HTML reporting plug-in for TestNG, known as ReportNG. ReportNG was the obvious way to go just because of its simplicity and easiness in integration with TestNG which is our testing framework. ReportNG provided us with a html report presenting each @Test with a pass / fail state.

Although ReportNG presented an excellent solution to our reporting problems it became apparent that for debugging purposes as well as for defect evidence purposes an extension to include screenshots was needed. These screenshots should be attached to every @Test annotation which fails along with the assertion failure.

In this post I will present you how we implemented the enrichment of ReportNG with screenshots taken by using the Selenium RC api.

ReportNG was used simply by included the dependency tag in the pom.xml of our Project, as:

<dependencies>
   <dependency>
     <groupId>org.uncommons</groupId>
     <artifactId>reportng</artifactId>
     <version>${reportngVersion}</version>
     <scope>test</scope>
 <exclusions>
   <exclusion>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
   </exclusion>
 </exclusions>
    <dependency>
</dependencies>

Selenium provides a function which capture a screenshot of the current window of the browser. We depend on this and we implement a function in Java which captures the screenshot and places it in the corresponding path in the filesystem as:

public void createScreenshot(ITestResult tr) {
 String testName=parseTestCaseName(tr.getTestClass().getRealClass().getName());
 String imagePath="screenshot-" + testName + "-"
                    + (new SimpleDateFormat("yyyyMMddHHmmss").format(new Date()))                    + ".png";
  try {
   Selenium sel = (Selenium) tr.getAttribute("selenium");
   log.info("Parameter selenium is set to :" + sel);
   if (sel != null) {
    String base64Screenshot = sel.captureEntirePageScreenshotToString("");
    byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
    FileOutputStream fos = null;
    try {
     File screenShotFname = new File(currentPath,imagePath);
     fos = new FileOutputStream(screenShotFname);
     fos.write(decodedScreenshot);
     log.info("Stored screenshot in file "+screenShotFname.getAbsolutePath());
     reportLogScreenshot(screenShotFname);
    } finally {
     if (null != fos) {
      fos.close();
     }
     } else {
      log.warn("selenium object is not set, cannot get screenshot");    
      log.warn(new Exception("STACKTRACE"));
     }

    } catch (Exception e) {
     e.printStackTrace();
    }

Ok, now with this function we have the screenshot stored in the corresponding path then we need to draw it into the reportNG log,the implementation is the following:

protected void reportLogScreenshot(File file) {
      System.setProperty("org.uncommons.reportng.escape-output", "false");
  
      String absolute = file.getAbsolutePath();
      int beginIndex = absolute.indexOf(".");
      String relative = absolute.substring(beginIndex).replace(".\\","");
      String screenShot = relative.replace('\\','/');
  
  
Reporter.log("<a href=\"" + screenShot + "\"><p align=\"left\">Error screenshot at " + new Date()+ "</p>");
Reporter.log("<p><img width=\"1024\" src=\"" + file.getAbsoluteFile()  + "\" alt=\"screenshot at " + new Date()+ "\"/></p></a><br />"); 
}

HINT: By default ReportNG escape the special characters, so in order not to see the href link in the reported log instead of your screenshot you will need to disable this escaping as:

System.setProperty("org.uncommons.reportng.escape-output", "false");

Voilà, and now the report log print the capture screenshot and display it as:

ReportNG

For the above implementation to work one should encapsulate it in a TestNG listener which calls these functions when the Test fails. We intent to release the aforementioned implementation along with the listener, as part of an upcoming open source release of our testing framework called Stevia. This framework will provide

• Selenium and WebDriver API unification: common API methods used for UI Testing, working for both Selenium and WebDriver!, with a flick of a switch 
• Enhanced verification methods providing element highlighting both in execution and reporting phases
• Enhanced ReportNG reporting with element highlighting and screenshot inclusion, both for Remote (Selenium Grid) and Local (Selenium RC, WebDriver) methods.
• Technical documentation, examples, and a lot more. 

Stay tuned for the official release and corresponding download link.... 

Read More

Setup Groovy in Eclipse

Groovy came in my Test Automation life here in Persado in order to make my daily life more easy and also make my coding more exciting. In this post I will present the way Groovy became part of my IDE, in order to start "groovy"ing.

Firstly, I installed the eclipse Groovy plugin, this was a piece of cake since I was only a few of steps away of it. Via the Marketplace of my Spring Tool Suite (STS): Help - Eclipse Marketplace - Search for Groovy -Install

Install Groovy plugin

Groovy plugin for eclipse can be also found under the following URL: Groovy Plugin for Eclipse

Since, the first step was completed, I moved to the second one and I created a new Groovy class by following the path in Eclipse: File - New - Other - Groovy - Groovy Class.

New Groovy class

  
My java code was already under a Maven project, so I followed the standard maven approach and placed my Groovy classes under src/main/groovy and src/test/groovy and keep my Java classes under src/main/java and src/test/java structure.

Maven Structure

HINT: In case the Groovy folders /src/test/groovy and /scr/main/groovy are not displayed, select your project and by right clicking select Build Path - Configure Build Path - Add Folder then include the folders on your view as:

Display Groovy Folders

Then, as a third step, I converted my project to a Groovy one by just selecting my project then by right clicking on option Configure - Convert to Groovy Project.

Convert to Groovy

Moving forward, I thought that I need to adapt Groovy in my maven word in order to compile my Groovy classes.
In order to achieve it, in the pom.xml, I included the dependency tag for Groovy:

<dependencies>
   <dependency>
     <groupId>org.codehaus.groovy</groupId>
     <artifactId>groovy-all</artifactId>
     <version>${groovyVersion}</version>
   <dependency>
 </dependencies> 

also, i needed a Groovy plugin with the goal “generateStubs”,so included the following tags:

<plugin>
  <groupId>org.codehaus.gmaven</groupId>
    <artifactId>gmaven-plugin</artifactId>
      <configuration>
       <providerSelection>1.8</providerSelection>
      </configuration>
        <executions>
         <execution>
           <goals>
           <goal>generateStubs</goal>
           <goal>compile</goal>
           <goal>generateTestStubs</goal>
           <goal>testCompile</goal>
           </goals>
         </execution>
       </executions>
</plugin>

The goal "generateStubs" was I a surprise for me when I realized that, as its name dedicate,creates java stubs of groovy classes.To understand it, consider a java class which needs a Groovy one to compile with and since maven standard compile process starts with /main and /test java packages and then with the /main and /test Groovy packages, compilation would stop with errors without it.

Finally, on my maven project i run Maven target "clean Install" to install all the dependencies and compile my source Java and Groovy code and i was ready to start rocking with Groovy…..That's all folks!!

Read More

Groovy up your test scripts

After one and a half year of developing test scripts using java a colleague in Persado convinced me to give a shot to groovy. Playing around for couple of weeks and there you are; I was convinced that groovy would be the new exciting thing in my testing life. Taming my excitement I decided to strategically replace blocks of java code with groovy. The plan was not to replace code in page objects neither test scripts but rather code in the supporting functions in abstraction layers between the test scripts and the page objects.
The decision in replacing the particular pieces of code was driven by the way groovy handles Lists, Arrays and Maps. In addition the effortless way of creating and using objects in a less wordy way than java showed the way.
Setting up groovy in my test environment was easy and I will show how in a feature post. With my environment ready and iron maiden in my youtube I started. First stop an object creation to carry data to a JMS dispatcher there you are.

import org.joda.time.DateTime
class MyObject {
String myString;
List myList;
Date myDate;
}

kaboom no setters no getters that simple how not to love the simplicity especially if your are not a developer. Now lets use the object, fill it up with data.

def rangeEvents = (11223300000 .. 11223300500).collect { it.toString()}
ldt = new DateTime()
def myObject = new MyObject(myList:rangeEvents,myString:"Hello",myDate:ldt);

Point 1: The list kind was never declared in the object definition.

Point 2: Assignment is as easy as myList:rangeEvents

Point 3: Ranges are Lists and each element of the list can be manipulated by a single statement
“list.collect { it.toString()}”

Of course it is expected that there would be people asking why go with groovy while I can do the same things with java ? Yes using groovy will not add extra arrows in your quivers but it can simplify your code and save you a lot of extra lines of code. Let me give you an example.

In one of my tests I wanted to simulate the following scenario

“There are 200 people owning a cat, 300 owning a dog 140 owning a lion and 230 owning a tiger. From the people owning the cat I want 15 of them to sent a letter and likewise 63 letters from the dog owners 22 from the lion owners and 33 from the tiger owners.”

The code I had developed with java spanned in several lines

int catLetters=0;
String[] catOwners = new String[15];
int dogLetters=0;
String[] dogOwners = new String[63];
int lionLetters=0;
String[] lionOwners = new String[22];
int tigerLetters=0;
String[] tigerOwners = new String[33];
for (String data : petOwners) {
   if (data.getOwner().equals(“cat”) && catLetters<15) {
      catOwners[catLetters] = data.getOwnerName()
      catLetters = catLetters + 1;
   }
   if (data.getOwner().equals(“dog”) && dogLetters<63) {
      dogOwners[dogLetters] = data.getOwnerName()
      dogLetters = dogLetters + 1; 
   }
   if (data.getOwner().equals(“lion”) && lionLetters<22) {
      lionOwners[lionLetters] = data.getOwnerName()
      lionLetters = lionLetters + 1;
   }
   if (data.getOwner().equals(“tiger”) && tigerLetters<33) {
      tigerOwners[tigetLetters] = data.getOwnerName()
      tigetLetters = tigerLetters + 1;
   }
}

Now with groovy we can create a Map with owners and expected number of letters lettersOwnersMap

def petOwnersList = []
requestedMessageResponses.each {
   for(String data : petOwners){
      if(lettersOwnersMap(it.key) && it.value>0){
         petOwnersList.add(data.ownerName)
         it.value--
      }
   }
}

Point 1: With the java implementation the code is static meaning that if we want to add a pet owner we need to add extra if statements. In the groovy implementation we only need to add an extra map entry that simple.

Update: In this post, check how to integrate Groovy in your tests while working with Eclipse.

Read More

Adding the Sizzle CSS Selector library in Webdriver

Selenium Webdriver has a weird opinion on Sizzle. Although (correct me if wrong) in the older Selenium-RC Sizzle was extensively used, in Webdriver it is only injected if the browser does not support native css selectors (see here). This is a pain if you've learned to love Sizzle's ability to "extend"  the normal CSS selector lingo with more advanced spices, and you're mainly working with Firefox, for which Webdriver uses native css.

History

Some background: In our little "ecosystem", we have lots of Selenium-RC code. And when saying lots, we mean LOTS. We are actively thinking to (at some point) take most of it and write it using Webdriver API, but the Sizzle-Webdriver saga is getting in the way. Or better said, was getting in the way.

Overriding Webdriver API

Webdriver uses the "By" class to encapsulate all possible ways of locating an element. This shows nice in your code, because you can have nice little snippets that make sense, e.g.

ExpectedConditions
   .visibilityOfElementLocated(
      By.cssSelector(
        "css=a.confirmation_link:not(.hidden)")) ;

Of course, the above is a Sizzle CSS locator so it would not work in native CSS. Bummer. Studying the Webdriver API for "By" (are you not? check here), we see the class itself is a holder for implementing classes of itself - a nice escape from the typical programming practices. We decided that this class should support Sizzle, and to do so, we extend it, initially to add our hook in it's cssSelector method:

public abstract class ByExtended extends By { 
public static By cssSelector(final String selector) {
   if (selector == null)
         throw new IllegalArgumentException(
           "Cannot find elements when the selector is null");
   return new ByCssSelectorExtended(selector); 
}

See our ByCssSelectorExtended ? (Yes, we're fond of large class names). We've basically extended the inner class inside By, named ByCssSelector, to do our magic. How? Well, basically we used the existing code, only to check if it actually matches something. The code has a nice "habit" of throwing an exception if it does not, so we cleverly "wrapped" it to do our biddings (for brevity, I only give you the code for the 1st method ;):

public static class ByCssSelectorExtended extends ByCssSelector {
  private String ownSelector;
  public ByCssSelectorExtended(String selector) {
    super(selector);
    ownSelector = selector;
  }

  @Override
  public WebElement findElement(SearchContext context) {
     try {
       if (context instanceof FindsByCssSelector) {
         return ((FindsByCssSelector) context) 
           .findElementByCssSelector(ownSelector);
       }
     } catch (InvalidElementStateException e) {
       return findElementBySizzleCss(ownSelector);
     }
     throw new WebDriverException(
       "Driver does not support finding an element by selector: "
       + ownSelector);
  }
  // ... ommitted code ...
}

Key here is our use of InvalidElementStateException - it is the exception thrown by Webdriver if the CSS cannot be located. We're not dealing with the other possibilities thus trying to make this extension function as the original in all other cases.

Our solution

Hope you're not tired so far... here go the juicy parts:

• we want to try the failed css locator with Sizzle (you may want the opposite, Sizzle first then native CSS)
• we want to check if Sizzle exists in the page, if not inject Sizzle so this may work
• finally, we want to try via Sizzle the failed css locator.

To do so, we want two helpers: isSizzleLoaded() and injectSizzleIfNeeded(), both using the Webdriver API to execute stuff via the JavascriptExecutor. I leave these to the readers' imagination and Google Search. Having those, the method findElementBySizzleCss is as simple as:

public WebElement findElementBySizzleCss(String cssLocator) {
  injectSizzleIfNeeded();
  String javascriptExpression = "return Sizzle(\"" +cssLocator + "\")";
  List elements = (List) 
       ((JavascriptExecutor) getDriver())
       .executeScript(javascriptExpression);
  if (elements.size() > 0)
    return (WebElement) elements.get(0);
  return null;
}

In conclusion, going back to our original example, here it is using our "Extended" version of By:

ExpectedConditions
   .visibilityOfElementLocated(
      ByExtended.cssSelector(
        "css=a.confirmation_link:not(.hidden)")) ;

And that's it. Voila!

Afterthoughts

We have given an example of how to override the default Webdriver behaviour towards locators - we are using this as part of our greater API that unifies Webdriver and Selenium-RC to produce a global/common access API for our QA/developers. For the specific "extension", we have created a "wrapper" function that hides our extending of By; all our locators go through that. This is another pattern we generally use - more on that on a subsequent post. Off for now...

UPDATE: We have released our API as open source! Check here and on GitHub! The full solution is there for you to find!

UPDATE #2: The By* extensibility for Sizzle seems very popular! To keep you up to date, our current 0.61.0-SNAPSHOT of Stevia now contains a very important fix for our method:
It was discovered by our users that when a wrong Sizzle selector is provided (one that is syntactically wrong) the Sizzle engine returns null. However, when trying to log this via an exception logger, Webdriver receives an NPE trying to output which selector and type was used; we never did set this up! The exception logged is the  NPE and not the actual exception about the missing/deformed selector, causing confusion. To fix this, we went as true commandos, since the API was protected.

private void fixLocator(SearchContext context, String cssLocator,
    WebElement element) {
 if (element instanceof RemoteWebElement) 
 try {
  Class[] parameterTypes = new Class[] { SearchContext.class,
   String.class, String.class };
  Method m = element.getClass().getDeclaredMethod(
   "setFoundBy", parameterTypes);
  m.setAccessible(true);
  Object[] parameters = new Object[] { context,
   "css selector", cssLocator };
  m.invoke(element, parameters);
 } catch (Exception fail) {
  //NOOP Would like to log here? 
 }
}

As you can see from the extract of code above, we have to modify the accessibility of "setFoundBy" so we can invoke it. After we do that, the call (using the Reflection API) will succeed and hence, no NPE anymore. Enjoy!!! - The actual commit is over here.

Read More

Missing Page Factories in Selenium RC ? Autowire

One of the new features introduced in selenium 2 was the usage of the page factories to easily instantiate a page object. For those who still use selenium and want to have a feature as the aforementioned one the solution is to use the spring source autowire functionality.

Although spring is a powerful framework for java enterprise development, testers that are usually not familiar with the dependency injection model seem to avoid it. The usage of the spring source Autowire feature can help to instantiate your page objects in an easy and efficient way.

In order to start Autowire you need to declare the instantiated a bean (page object) to src/test/resources/applicationContext-test.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:context="http://www.springframework.org/schema/context"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
       http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">
 <context:annotation-config />
 <context:property-placeholder location="classpath:mcs-config.properties" ignore-resource-not-found="true" system-properties-mode="OVERRIDE"  />
 <bean id="firstPageObject" class="com.mycompany.FirstPageObject"/>
</beans>

After the bean has been declared just use the Autowire annotation inside the script to instantiate the page object

@Autowire
FirstPageObject firstPageObject

Likewise for actions resulting to new page objects (click, select) introduce a method returning the resulting page object and Autowire the resulting page object

public FirstPageObject myMethod(){
   return firstPageObject;
}

Read More

Maintainability 101

One of the biggest problem in the automation script development is the script maintenance. Even that there are a lot of widely accepted development methods for improving script maintenance, there are misconceptions that can easily direct testers in writing unmaintainable scripts.

The most popular methods in developing maintainable test code is the Page Objects pattern. Although using the Page Objects pattern will lead in a more maintainable code, the misuse of the pattern will only result in no more than adding an abstraction layer between the page under test and the test script each self. The misunderstandings in applying the pattern in order to create more maintainable test scripts come in the usage of the objects locators. The proper usage of the objects locators would result in the preferred maintainability and not the usage of the pattern. Taking into account that our object locators are locally declared (I totally agree with the analysis of Adam Goucher in his post http://element34.ca/blog/the-great-locator-location-debate) a poorly implementation of the pattern could result in using the page locator in more than one place inside the page object. For example having an input field for which we need to implement an object method to type in a value and an object method to read the inputed value we should use a locally declared locator as a static string instead of having a locator declaration in each method.

public void typeInputValue(Stringvalue){  
    selenium.type(“//input[@name='x']”);
} 

public void readInputValue(){ 
   selenium.getValue(“//input[@name='x']”); 
}

public static final MY_LOCATOR=”//input[@name='x']”;

public void typeInputValue(String value){ 
   selenium.type(MY_LOCATOR); 
} 

public void readInputValue(){ 
   selenium.getValue(MY_LOCATOR); 
}

In the first example a change in the objects locator would result in updating the object in to two places inside the page object rather than in one place as shown in the second example.
One of the goals in test script development is to make our test code more robust in locator breakages. As easily understood the single appearance of object locators in the page objects leads us toward this goal.
At this point I would like to thank Adam Goucher for his wonderful and extremely useful posts regarding test automation. For all it worth I would like to say that his posts lead the way in changing our test scripts to a more robust more maintainable test scripts.

Read More

Selenium WaitFor ...

One of the most difficult problems to overcome when developing automated test scripts is time synchronization. Black box testing is based on expecting specific results upon a specific user actions irrespectively of time. Thus the big question that comes in mind is “when to expect the results?” are the results produced instantly or it takes some time depending on the system load?

One of the solutions presented, is using the time delay available to the programming language used for the test script development. Using Selenium with Java Thread.sleep(“3000”); will do the job. But is this effective? If you are developing automated test scripts for small scale projects and time is not an issue maybe this is your solution; but its advisable to use time delays large enough to account for system slow downs under load.

In large scale projects static time delays are inefficient. Think the amount of time wasted when  hundreds of scripts, having  static time delays, are executed. For these cases the solution to time waiting problem is to use a dynamic waiting period. Dynamic waiting period is the time duration until the system reacts. Testing a web application the reaction of the system would be a page loading. For these cases selenium.WaitForPageToLoad(“3000”); comes in handy. Using WaitForPageToLoad command insures that the system reacts before results verification. Again in this process it is critical to allow large amount of delays to account for system slow downs due to overloading. In our tests it was observed that sometimes a page would load in 10 seconds and under load the page loading could take up to 1 minute, so we adjusted time delays to the observed maximum. The debate in using the WaitForPageToLoad method is what is an acceptable waiting period for a page to load? Is it 1 ,2 ,3 minutes? In debates like that I always like to consider the criticality of the system and the defined requirements for the time responses. In non critical systems with no time requirements defined, it is acceptable to use large delay times (do what you can for the functional tests to complete and the performance tests will optimize the system to react faster under load). In critical systems usually time reaction requirement are well defined so time delays can be extracted through them.

But what happens in systems implemented with Ajax were elements are dynamically refreshed without a page loading? Then the selenium.WaitForPageToLoad(“3000”); it is to no use. In these cases prefer to use the selenium.waitForCondition(); command. When I came up to this situation most of the examples I found to help me were referring me in using JavaScript with the WaitForCondition command. Although this was feasible, using Javascript would have increased the difficulty scale of test script development (taking into account that testers are not required to be experienced programmers) I preferred staying to selenium commands and Xpath references to find dynamically refreshed elements. In some cases I replaced the selenium.WaitForPageToLoad(“3000”); with the WaitForCondition statement in order to have more control into what I am expecting to be loaded.

So a selenium.WaitForPageToLoad(“3000”); can be replaced by

selenium.waitForCondition("selenium.browserbot.getCurrentWindow().document.getElementById('page-heading').innerHTML='New Offer'","10000");

In the above example the waiting condition is specifying which page load to expect according to the page heading. Obviously using the  WaitForPageToLoad statement could result in navigating to the wrong page something that could be avoided using the WaitForCondition statement.
In the above  WaitForCondition example I used the DOM to access the page heading and in the following example I will show how to use the  WaitForCondition using the Xpath

selenium.waitForCondition("selenium.isElementPresent(\"//div[@id='cboxContent']\");", "10000");

In the above example we wait until an element is present (The element's Xpath is specified by the id).

In conclusion WaitForCondition offers better control over the aforementioned solutions controlling what to expect and when thus resulting in test robustness.

Read More