Model based testing (MBT) is a test design method where we start from a model of the planned code and the test cases are generated from the model. Furthermore, even executable tests can be generated from the original test cases. Experts strongly believe that MBT will used widely in 5-10 years.

4Test is a new MBT method where a test design method Constraint Driven Testing (CDT) and a test automation method Keyword Driven API Testing (KDAT) are united.

4Test is a model based testing method where the elements of the model is a triple: {category, choice, constraint}. The goal of 4Test is to generate executable tests for test automation tools.

We can use the well-known Gherkin syntax for the description of the constraints. The original syntax is extended and an entire constraint can be written in one line. Here is an example:

WHEN card owner IS yes AND new books’ price IS 49.99 THEN price reduction for new books IS 10%

Constraints are understandable for human and machine as well.

KDAT can easily be integrated with CDT. The reason is that in case of KDAT the tests do not contain any implementation related elements such as click on here and that. Similar is the case of CDT. As a result the generated test table of CDT can easily be converted to be readable for any test execution tool. Fortunately, modern test automation tools involve keyword driven testing.

CDT and KDAT are ideal methods to be united. The connection among them is straightforward. The features are mapped to keywords which represent method/function calls. The input categories are mapped into parameters of the keywords. The output parameters are the return values of the methods/functions. The keyword hierarchy shows the program structure.

4Test is provable better than widely used flowchart based methods (e.g UML) as it needs fewer tests and has a more reliable test set.

Test Automation

Software systems become more and more complex while time to market is being shortened all the time. Program modifications and bug fixing affect other parts of the code, causing new bugs to be introduced into the systems. It’s impossible to test everything manually prior to each release. The solution is test automation, but inappropriate automation methods result in more issues than benefits.

4D Soft introduced and uses an advanced test automation method. The key element of test automation is maintainability. With respect to business logic, API testing is more stable than GUI testing since it avoids the problem of changing the GUI. Our company has introduced Keyword Driven API Testing – KDAT, by which coding work is reduced while testers only need to perform genuine testing work. We also make it possible to test applications written in almost any widely used programming language by applying our remote-keyword method.

GUI testing is necessary when we test the GUI itself. Our advanced and matured keyword-driven method makes it possible to write tests that are as stable and functional as possible. We use open source or proprietary tools as appropriate, but our users need to be aware that methods are more important than tools.

Our methods strongly support the agile methodology, but they can be used for non-agile methodologies too.

Mobile Testing

Mobile application testing is a process by which application software developed for handheld mobile devices is tested for its functionality, usability and consistency. We plan, implement and execute functional tests for Android, iOS and Windows 8 apps with the Ranorex professional test automation tool.

We are able to create highly reusable cross-platform tests if the application contains similar logic in both operating systems. We use object-based recognition that makes our test language independent and robust.

For similar test cases, data-driven testing is a supported option for testing a login functionality based on an Excel or SQL database, for instance. We can select testing devices from a data source in the same way, so that we can cover a great deal (over 80%) of device coverage on both supported platforms.

There is no need to modify (root, unlock) the testing device to access these features, the solution only runs on the tested application itself.

Web Testing

Web testing is the name given to software testing that focuses on web applications. Complete testing of web-based systems before going live, can help address issues before the system is released to the public.

We create and maintain cross-browser test cases for the most popular web browsers (Internet Explorer, Edge, Mozilla, Chrome, Safari). These test cases are built on robust – xpath-based – RxPaath object recognition. It is significant that we dont use numberized contents in the xpath, but instead employ evolved-type and relationship-based recognition. This method is also able to support dynamic IDs.

It is possible to capture and replay tests, for one-time usage, but we prefer to create more professional and maintainable solutions, so that the tests became more robust. We can run tests in parallel in several browsers simultaneously, and also can initate test runs form the most common CI systems (Jenkins, Bamboo). We create robust yet easy-to-maintain test cases, so that once a test case is done, the common parts are highly reusable.

Remote Testing

Remote testing is a method that can be used when the test tool and the system under test are on different machines. It can be used to test distributed systems where we write the test cases on our local machine where our testing tool is implemented and run the tests remotely. However, remote testing can also be used for simple local API testing.

An advanced method is Keyword Driven API Testing, KDAT, where keywords are the method names of the SUT, and these methods are called with necessary input parameters, while the return values are captured and validated by the testing tool. If the testing tool is able to recognize the methods as keywords, there is no need to know anything about the programming language. based on the specifications provided, the tester writes the test cases by applying keywords, which will be implemented by the developer. Prior to the test execution, the tester connects to the remote application. By applying the Remote Procedure Xall (RPC) protocol known as XML-RPC, the local testing tools are able to recognize the remote and appropriately annotated methods as keywords. Therefore, it is straightforward to execute our tests as the method calls are simply encoded as XMLs and sent. On the remote machine, these calls are unwrapped and executed.

The main advantage is that it’s enough to have one testing tool such as Robot Framework, and we can test applications written in different languages in the same elegant way.

Keyword Driven Testing (KDT)

Keyword-driven testing is a software testing methodology that separates test design from test development. Among other benefits, this allow additional professional groups, such as business analysts, to work on the test automation process. By applying KDT, test cases can be created without implementation issues and independently from the writing of test code. In this way, functional tests can be developed prior to any implementation work.

Keywords represent functional operations such as 'Open Browser' or 'Add a Selected Book to the Chart'. These keywords with appropriate parameters can also specify a direct call to the related test code. Keywords covering minimal operations can be collected into larger keywords. In this way, keywords can be structured and applied to different projects. Usable keyword libraries already exist, so there is no need to reinvent the wheel.

Besides separating test design and test automation, the main advantage of keyword-driven testing is that test suites are understandable for everybody. Because of this, the tests can be considered an executable specification as they are both human-readable and executable. Because of the structural properties of KDT, you will only need to modify a few test cases when the code is modified. Therefore, tests are stable and maintainable.

Constraint Driven Testing

The key to testing is to create the best test cases for a given application. An exhaustive test is impossible and yet testing strategies where the number of tests is not linear with the code is not possible in practice.

We can order tests based on their significance, and we can use boundary values and equivalence class testing. However, these methods will not ensure that the number of test cases remains near optimal. We need a systematic method called category partitioning by which very efficient tests can be created.

We improved the original method to reduce the number of test cases. We start from a user story. This story shouldn’t be too difficult, i.e. no more than six lines of text. Based on the story, we gain categories which are the preconditions, input and sometimes the output values. For example, based on the user “login”,story we can select login name and password as categories.

The categories are further divided into choices, for example a login name can be existent, non-existent and duplicated (person trying to login twice).

There are constraints on the choices, for example in the event of a non-existent login name, the password is irrelevant, and therefore will not be tested by this test case.

We can define a test data adequacy criterion with respect to this method: the story is tested when every constraint and every choice has been tested at least once. With this approach, we are able to efficiently test the user stories to ensure no superfluous and no missing test cases occur.

Category partitioning is a good starting point for keyword-driven testing, since choices and keywords can relate to each other.

Keyword-Driven API Testing: KDAT 

A recent Forrester Wave study validates the test automation pyramid, i.e. API testing should be increased as a GUI test expense:

We define API testing as a method where business logic is directly tested without execution of the GUI. However, with the exception of some special cases, the traditional solution requires a considerable amount of coding. What is more, testing knowledge is also required. In addition, this is not an agile method, so many teams continue to test through the GUI.

Our new Keyword-Driven API Testing or KDAT is a test-first agile method where the Graphical User Interface is replaced by a Test Tool Interface (TTI). A TTI is a piece of code that can send input to and receive output from the business logic. A GUI is a connection between the human and the business logic, while the TTI is a connection between the test tools and the business logic:

The method signatures of the Test Tool Interface can be exported as keywords. In this way, the necessary input for the business logic can be set by the test tool as keyword parameters, and no GUI is executed during the testing process.

The advantage of this method is that testing and coding work is separated and everybody is able to focus on what they do best. With the keyword-driven method, KDAT can be used in agile projects and the tests will be understandable for everybody. To learn more, watch our presentation.

4D Test Framework

A very basic test automation framework is a system that allows test suites to be executed, test reports to be displayed and test libraries to be easily incorporated. Robot Framework is in itself a framework. You can plug-in both your self-coded test libraries, as well as popular libraries such as Selenium2Library.Robot framework is a keyword-driven tool. By importing it into a test library, you can harness all the power of, Selenium, for example, by building tests that apply the appropriate Selenium keywords. Robot framework includes many test libraries such as AndroidLibrary, IOSLibrary, SSHLibrary, HTTP test library using Requests, and many more.

One of the key advantages of using RF is that you can write your test code and export test method signatures as keywords. RF has an IDE called RIDE, which helps you write and execute your keyword-driven tests. Therefore, you can write your test code in the same language as your core code, and RF can still use it.

RF works with build automation tools such as Maven. Since Maven is integrated into Eclipse and other IDEs, you can work without any changes and simply create your projects as Maven projects. Lots of test libraries have Maven dependencies, so you simply need to copy a dependency and Maven downloads the necessary files. This allows you to execute both your application and your tests from Eclipse. As Jenkins and Maven can also be easily integrated, you can schedule a test execution extending your CI process.

One of the greatest and most singular features of RF is its true remote test execution capability. In this case, the system under test can run on a remote server while the tests and the test tool are on our local machine. As the applied technology is the XML-RPC protocol, any code can be tested. We can use also this remote solution on our local machine. In this way, any single test case can be executed by RIDE, which would be very difficult otherwise.