Pairwise Testing

What is Pairwise Testing?

Pairwise testing is a software testing technique that focuses on covering all the different combinations of input values. It significantly reduces the number of test cases as it only tests every possible pair of inputs instead of considering each combination, making it one of the most efficient and rapid approaches to validate an application compared to exhaustive test methods.

How Does Pairwise Testing Work?

The process of applying pairwise testing can be divided into five steps. I will explain these five steps using a simple scenario. Consider an online store where customers need to select three options (country, currency, language) for each item before placing their order.

Country: US, UK
Currency: USD, GBP
Language: English, Spanish

Extensive testing will require eight test cases, while pairwise testing will only require four. So, let’s see how pairwise testing reduces the number of test cases.

Step 1 – Identify Input Parameters

First, you need to decide the parameters that affect the system’s behavior. In this case, the parameters are:

Country
Currency
Language

Step 2 – List Possible Values

Then, list all possible values for each parameter.

Country: US, UK
Currency: USD, GBP
Language: English, Spanish

Step 3 – Create all Permutations

Now, create all possible permutations between those three parameter values.

Step 4 – Create Pairs

Generate all possible unique pairs depending on the listed values. The pairs are:

US, USD
USD, English
USD, Spanish
US, GBP
GBP, English
GBP, Spanish
UK, USD
UK, GBP

Step 5 – Design Test Cases

Design appropriate tests so that each value pair is compared with every single permutation. In the table below, eight test cases can be generated based on these three parameters.

The last column shows whether a particular test case can be chosen depending on the pairwise testing strategy. For example,

Test case 01 has been selected because it covers the pairs (US, USD) and (USD, English), which are also covered in test cases 02 and 05.
Test case 04 has been selected because it covers the pairs (US, GBP) and (GBP, Spanish), which are also covered in test cases 03 and 08.
Test case 06 has been selected because it covers the pairs (UK, USD) and (USD, Spanish), which are also covered in test cases 05 and 02.
Test case 07 has been selected because it covers the pairs (UK, GBP) and (GBP, English), which are also covered in test cases 08 and 03.

As you can see, the pairwise testing approach has reduced the number of required test cases by 50% compared to exhaustive testing.

Advantages of Pairwise Testing

Pairwise testing is frequently used in software development for a number of reasons. Its popularity is primarily due to:

Reduction in test cases: As shown in the above example, pairwise testing significantly reduces the number of test cases.
Increase in test coverage: Although examining parameter pairs reduces the number of test cases, it addresses all the possible scenarios, ensuring almost complete coverage.
Time efficiency: Executing the test cases in a more extensive application takes a lot of time. Pairwise testing reduces the number of test cases and the time it takes to execute them.
Overall budget reduction: Reducing the number of test cases leads to a decrease in the overall testing budget.
Simplified test design: Pairwise testing simplifies the creation of test cases. Test cases can be generated methodically using automated pairwise testing generator tools like Allpairs, PICT, and Pairwiser.
Cost-effectiveness: Fewer test cases reduce the overall cost, making pairwise testing a cost-effective solution.

Disadvantages of Pairwise Testing

Even though pairwise testing has many advantages, it might not be the best option for testing in some cases:

Missed combinations: Pairwise testing might not discover issues related to specific combinations of three or more parameters. For example, in a banking application, defects may occur only with certain combinations of Account Type, Transaction Type, and User Location. However, pairwise testing may miss this scenario since it only considers combinations in pairs.
Diminished defect yield ratio: If a critical combination is missed, the defect yield ratio may decrease.
Additional testing required: Pairwise testing alone might not be enough for complex systems since it doesn’t cover all the possible scenarios. You might need to use additional testing methods in such situations to cover other test cases, including edge scenarios.
Time-consuming process: For some scenarios, pairwise testing might not be the fastest solution even if the number of test cases is decreased. For instance, creating and executing pairwise test cases can take a significant amount of time for a large enterprise application with many input variables.
False assumptions: Pairwise testing assumes that most defects are caused by interactions between pairs of parameters, which might not be the case for many real-world scenarios.
Overfitting to test cases: Sometimes, pairwise testing programs can be overfitted to pass the test cases. This will cause the application to fail in real-world scenarios, although it passes all the test cases.