The key to testing is to create the best test cases for a given application. Test case creation and implementation is both costly and time consuming, meaning that typically the goal is to optimize this process by planning a moderate-sized and efficient test set which reveals most of the bugs.
However, testing itself is not enough: according to a huge amount of collected data published by Capers Jones in 2012, on average only about 85% of bugs are found in the testing stage. Defect prevention in IT is as important as disease prevention in healthcare.
There are various methods available, from formal inspection to static analysis. Applying these methods can result in significant improvement, with significantly fewer bugs going undetected.
Constraint Driven Testing (CDT)
Our method originates from category-partitioning, published by Thomas J. Ostrand and Marc J. Balcer in 1988. We have sought to establish a widely applicable test design method which:
- optimizes the number of test cases
- introduces a test data adequacy criterion to determine when to stop creating additional testing
- adapts the method for agile testing
Although it is possible to use boundary value and equivalence class testing, etc., these methods do not ensure an optimal number of test cases. The solution is CDT, a systematic method which enables us to create very efficient tests.
We start from a user story/feature specification. This feature shouldn't be too complicated, i.e. no more than six lines of text. If the feature is too complex it should then be split into smaller ones. We then analyse the feature and try to divide it into categories related to preconditions, inputs, and sometimes outputs. For example, for a login feature we can select login name and password as categories.
The categories are then further divided into choices: for example, a login name can be either existent or non-existent.
There are constraints among the choices: for example, in the case of a non-existent login name the password is irrelevant, and will not therefore be tested by this test case. Another constraint is that you cannot select a password which is the same as your login name.
Now, if we have all these things together, we can start creating the test specification. A test specification contains test cases where sometimes direct values may substituted by choices, i.e. login name as "existent" instead of "Smith". Test specification creation must start by satisfying all the constraints.
When these constraints are processed we then make or generate a table, where the columns are the categories and the rows contain the tests with the selected choices:
The final step is to replace the choices with actual values.
New test data adequacy criterion
A significant advantage of this method is that it is possible to define a suitable test data adequacy criterion. This criterion helps us to decide when to stop creating additional tests. The criterion here is very simple: each choice has to be selected at least once. However, if there is a constraint stating that choices are irrelevant at a given test case/specification, then no selected choices are considered to have fulfilled the criterion.
When making the test frames and selecting the categories, choices and constraints, any incompleteness, ambiguity or inconsistency is easily revealed. This is an excellent review method, since the specification is deeply processed and analyzed for test creation, whereas no simple or systematic reading would be able to reveal the deep problems contained in the specification.
Another source of defect prevention is the test first method itself. Good and readable test cases drive developers to make better code.TDD is also considered to be a good method with 37% defect prevention efficiency, and since CDT results in optimized tests, we can assume a higher prevention rate when considering only the test first element of prevention.
When to use?
CDT can be used in every test design process to create test cases for the latter test implementation phase, and even for unit and component testing. Testers can create the test cases from the functional specification/user story and the developers can then implement them. The method can also be used for keyword-driven testing, when categories (and sometimes choices) can be considered as keywords.
However, the biggest advantage is that CDT and our other method KDAT can be united into a complex model based testing method referred to as 4Test. 4Test can be automated resulting in the generation of executable test cases.
Price of food delivery in Budapest
The price of food delivery is HUF 600. In central districts (V., VI. and VII) delivery is free. If the price of the food reaches HUF 6000, then delivery is also free. The lower limit of food delivery is HUF 1000.(HUF 6000 is between USD 20 and EUR 20. Restaurants in Hungary are really cheap, let's visit Budapest)
Consider the categories based on the specification. These are the location from where the food is delivered, the price of the food, the price of delivery and whether the price has reached the limit, or no.All these categories are implicitly or explicitly mentioned in the specification (see the yellow text).All other elements of the specification is related to choices or some explanations.
Note that only some of the choices are in the specification, the others are selected to fulfill the requirement of boundary value testing. In this way we select choices of Price of food as 990 and 5990. Here is the table for categories and the related choices:
Categories with choices
The constraints are constructed based on the specification and contain the categories and the choices.
- If Location = central, then Price of delivery = 0
- If Price of food >= 6000, then Price of delivery = 0
- If Price of food < 6000 and Location = non-central, then Price of delivery = 600
- If Price of food < 1000, then Reach delivery limit = no
After finishing the constraints we mark with (S) those choices which are to be selected only once. For example, if we selected central, which implies free delivery, then selecting it again would hide the real testing of the case when the food price would cause the free delivery.
Now we can create the test specification (test table) below.
We can see, that all the choices have been selected at least once, therefore the testing criterion has been satisfied and no additional tests have to be selected. Our hypothesis is that we are able to select constraints to cover all the choices for any specification. However, it is not necessary, since the missing tests can be inserted easily. In the practice only one or two additional test cases are required.
With this method, developers can avoid test design leaving it to professional testers who are experts in the field. The tests are fully optimized, since no missing or superfluous tests will be created, assuming that both the feature description and the application of CDT are correct. Maintenance of test specification is very easy and totally independent from both the implementation and the selected testing tools.
The method involves a test data adequacy criterion. This is much better than code coverage criteria, since it can be used prior to coding, it is unambiguous and it can always be fully (100%) satisfied.
The method also aids defect prevention at no additional cost, by simply following the necessary steps in order to produce the test specification, meaning that it is an efficient way to unite defect prevention and test design.
CDT is clearly agile, though it can be used for other methodology as well.