In our agile project we utilized regression testing with
test automation and continuous executions of our automated test scripts. This
process served us well while the test execution lasted less than 4 hours but
started to create major problems when the total number of test scripts
increased and the execution time exceeded 1 day. A quick relief to our problem
came with introduction of parallel execution of test scripts but still in cases
were the management wanted a quick answer to “Do we deploy on production or
not?” it wasn’t enough.
One way to design effective regression suites, to ensure the
continuation of business functions, is to follow the rule:
Regression test suite!= sum(Functional test cases)
The purpose of the functional tests is to explore the
behavior of specific business functions and highlight corner cases while the
purpose of regression tests is to give an overall overview of the entire
system. In this way the design of the regression suite should include test scripts
inspecting the end-to-end business functions that a system encapsulates.
For the purpose of our project (agile environment) the
regression suites were designed with the hypothesis that the GUI, boundary and GUI
negative tests should be separated from the runtime tests because:
- Changes in the code in boundaries are rare
- GUI is indirectly tested (application its self is used for the preconditions of each test case)
This separation leads us automatically in designing
different regression suites containing these special categories.
More over, a vertical separation according to execution
frequency is needed. In an agile environment in a 2 weeks sprint, 4 sprints
release circle, effective execution regression times could be at the end of
each sprint, in every other sprint and at the end of the release. The schedule
of each regression suite could be decided according to how often this part of
the code is changed. For example a boundary test is highly unusual to change
thus a planned execution time could be only once per release and preferably one
sprint before the release sprint (latest spring
usually reserved for sprint new features).
As mentioned before at the end of each sprint the new
features suite should executed additively. For example in sprint 1 we have 5
features and in sprint 2 we have 3 features. The regression suite should have
cases testing the 5 features (end of sprint 1) and cases 8 features (end of
sprint 2)
