Thursday, August 29, 2013

Regression Suites

In our agile project we utilized regression testing with test automation and continuous executions of our automated test scripts. This process served us well while the test execution lasted less than 4 hours but started to create major problems when the total number of test scripts increased and the execution time exceeded 1 day. A quick relief to our problem came with introduction of parallel execution of test scripts but still in cases were the management wanted a quick answer to “Do we deploy on production or not?” it wasn’t enough.

One way to design effective regression suites, to ensure the continuation of business functions, is to follow the rule:
Regression test suite!= sum(Functional test cases)
The purpose of the functional tests is to explore the behavior of specific business functions and highlight corner cases while the purpose of regression tests is to give an overall overview of the entire system. In this way the design of the regression suite should include test scripts inspecting the end-to-end business functions that a system encapsulates.

For the purpose of our project (agile environment) the regression suites were designed with the hypothesis that the GUI, boundary and GUI negative tests should be separated from the runtime tests because:
  1.      Changes in the code in boundaries are rare
  2.      GUI is indirectly tested (application its self is used for the preconditions of each test case)
This separation leads us automatically in designing different regression suites containing these special categories.

More over, a vertical separation according to execution frequency is needed. In an agile environment in a 2 weeks sprint, 4 sprints release circle, effective execution regression times could be at the end of each sprint, in every other sprint and at the end of the release. The schedule of each regression suite could be decided according to how often this part of the code is changed. For example a boundary test is highly unusual to change thus a planned execution time could be only once per release and preferably one sprint before the release sprint (latest spring usually reserved for sprint new features).

As mentioned before at the end of each sprint the new features suite should executed additively. For example in sprint 1 we have 5 features and in sprint 2 we have 3 features. The regression suite should have cases testing the 5 features (end of sprint 1) and cases 8 features (end of sprint 2)