In one of the projects I am involved with, common functionality applies to various communication channels such as Sms, Email, Ad-words and Web banners. Naturally in a situation like this the question comes in mind is: "Do we repeat the test cases for every communication channel?" The answer I am always giving is: "It depends on the qualification method used".
For qualification methods such as Manual or Exploratory testing it’s a matter of time and manpower to execute as many times as our communication channels or use pair wise to optimize testing effort while minimizing risk of missing defects. For qualification methods such as Automation testing in order to achieve the the highest coverage we execute as many times as our communication channels.
Automating the test cases can be done in two different ways. The first way is to have four different test scripts differing the channel. This is bad practice because of maintainability issues. The second way of automating is to use a predefined TestNG annotation to execute x times a predefined test script.
My initial approach in using predefined TestNG annotations was to use data providers. The problem in this approach was that I needed to set up the communication channel in the @BeforeClass section and the data providers should be used in @Test section of the test script.
The second approach in using predefined TestNG annotations was to use the Factory functionality. The problem with this approach is that TestNG does not respect the dependency of each test step in a single thread. BeforeClass will be executed as many times as specified in the factory then Test Step 1 as many times and so on so forth in a single web session.
One fitted solution, out of the predefined TestNG annotations scope, is to create a base class implementing the test. The base test should hold the aforementioned communication channels as a parameter. The parameters should be passed in an TestNG parameter annotation in @BeforeClass section and set before the execution in the testng.xml parameters section.
@BeforeClass(alwaysRun = true)
@Parameters(["test.channel"])
<parameter name="test.channel" value="Email" />
For maintainability and traceability purposes I have created 4 different empty test scripts (traceability) extending the base test class (maintainability).
This scenario is ideal if the functionality of the software is identical for every communication channel, but if there is a slight change in a test step what do we do? One easy solution is to skip the test step. To do this in TestNG use SkipException. Keep in mind in the reporting the skipped test steps shown as skipped, and this can be confusing in cases there are failures in the @BeforeClass section. A more clean approach to skip the test step will be to set the @Test(enabled=false) flag of the annotation, but we need to dynamically change its value in the context of a test suite. This can me accomplished by using the ITestAnnotation provided by TestNG like:
public void transform(ITestAnnotation annotation, Class arg1, Constructor arg2, Method testMethod) {if(campaignTestChannel.equals("Email")) {annotation.setEnabled(true);}}
The above piece of code implements a functionality that if the channel is the Email all test steps containing enabled flag should be executed.
As a conclusion given the aforementioned situation in which multiple script execution is needed the automation is the way to go. To achieve maintainability avoid replicating test scripts and choose one of the methods (data providers, factories or extending a base class) that fits your needs. If you need to exclude test steps from your base class and don’t care about reporting go for the skipExceptiuon method, other wise dynamically on runtime transform your annotations.
