 | Level: Advanced Randy Masciana, Testing Consultant
13 Apr 2005
Updated 11 Nov 2005 IBM Rational Functional Tester 6.1 can empower testing staffs to create modern, pattern-based, reusable testing frameworks. This advanced-level article shows you how.
Editor's Note: This is a very technical article for the advanced user. The approach outlined in this article should be used in context. It should not replace requirements-based testing, but supplement it. This approach is very powerful and can generate a significant volume of tests. The quality of any given test depends on its ability to validate a realistic use case or requirement.
The case for a reusable testing framework
The test design pattern, Design an architecture for object-oriented component testing, created by Dr. John D. McGregor, states the following: "Often the basic architecture of the testing software can be designed to be beneficial across many applications and testing situations. The basic generic functionality can serve as a foundation for testing many different types of components and as the interface to standard tools." As with many test design patterns, this pattern at once illuminates an often-murky testing problem, namely that of repetition of testing code, and sheds light on a possible solution: create a reusable testing framework.
What would you need to create a reusable testing framework? At the core, you would need a layer of abstraction between the objects comprising the testing methodology (that is, as described in various test design patterns) and the objects under test (that is, as manifested by various GUI controls). Such abstraction would require the capabilities of a full-fledged object-oriented programming language ? one that is replete with robust reflection methods. You would of course need a technology that recognizes GUI controls and exposes their properties in an object-oriented fashion. Also required is an integrated development environment that is aware of all of the ways that the GUI control objects can be manipulated and verified. And finally, an open enough environment to allow the inclusion of a wide variety of ancillary tools would allow for the creation of the most effective framework.
A tool comprising all of these features would have to be easy enough to use that a testing professional could make a candid case to management that, were such a framework employed, time spent creating a reusable testing framework would result in an overall time-savings, when considering the testing time saved across all applications under test. IBM® Rational® Functional Tester 6.1 (see Figure 1), running in the Eclipse IDE, provides all of these facilities. Check out the iBM Functional Tester product pages for other features. Functional Tester comes in two flavors: VB.NET scripting, which runs inside the Visual Studio .NET IDE, and a Java™ scripting version, which runs in the Eclipse environment. The Java scripting version will be considered in this article. Because a product now exists to enable the creation of a reusable testing framework, there is no reason not to begin.
Figure 1. IBM Rational Functional Tester Version 6.1 (Functional Test Perspective)
A test pattern approach
Once you have the Functional Tester tool in hand, the fun of creating a reusable framework can start. But where do you start? How do you proceed? Are there any guidelines? Fortunately, some work has gone before you. In the spirit of other types of software design patterns, a number of test design patterns have been created. Of course, they do not cover everything needed to create a reusable testing framework, but it?s worth considering them. The Software Testing Patterns website reprises the benefits of using patterns:
- They provide a vocabulary for problem-solvers. "Hey, you know, we should use a Null Object."
- They focus attention on the forces behind a problem. That allows designers to better understand when and why a solution applies.
- They encourage iterative thinking. Each solution creates a new context in which new problems can be solved.
Requirements for a reusable testing framework
So, we have the tool (Functional Tester) and we have the design approach (use patterns when applicable). It?s time to review requirements. We?ll take for granted the many requirements that Functional Tester meets "out of the box" and concentrate on generic capabilities that we have to add to create a reusable framework. The goal is to design a testing framework that enables the following new functionality:
- Automated test input generation
- Automated production of expected results (an oracle)
- Automated comparator
These capabilities are taken
Testing Object-Oriented Systems Models, Patterns, and Tools
, by Robert V. Binder. Many more requirements are listed in the book, however, the rest of them are either handled automatically by Functional Tester or are not required for a generic solution.
Automated test input generation
In the ?out-of-the-box? implementation of Functional Tester, scripts are recorded, and then the tester performs the following tasks: modularize the scripts, data-drive the desired inputs, create desired verification points, and then play back the script. Problems can arise, however, if the number of test cases becomes large. In this case, the time required to record the steps for all of the test cases might be unfeasible. This problem becomes magnified when multiplied over many testing projects. The solution is to provide a generic framework for generating test inputs that can be reused across all testing projects.
There are three main methods for automatically generating test input:
- Functional Tester datapools (which can be imported from IBM® Rational® TestManager datapools)
- Simple, programmatically controlled XML or relational database getting and setting
- Programmatically controlled XML or relational database getting and setting, augmented by algorithmic generating and filtering.
Each method has a valid role in a generic testing framework.
Method one: Input information using Functional Tester datapools
Functional Tester datapools, is useful for generating variable data that does not have to derive from complex algorithms (for example, Address, or Phone fields). You can take advantage of the auto-formatting feature of IBM Rational TestManager datapool creation to create simple patterns like social security numbers, first and last names, addresses, and zip codes. Then, Functional Tester can import these datapools (See Figure 2).
Figure 2. Importing a datapool
After importing, you use Functional Tester to associate the datapool with your script(s). Functional Tester then allows you to select which items to substitute with datapool values. You can refer to datapool fields by index or by name in your script. After the reference has been made, the values can be programmatically altered if needed. Download the necessary code to insert datapool values at bottom of the article.
Method two: Input information by simple retrieval of XML or relational data
Simple, programmatically controlled XML or relational database getting and setting, is useful when only a trivial algorithm is required, but the data pattern is still too complex for datapool generated inputs. An example would be the auto-generation of patient account numbers or general ledger numbers in which some semantics are embedded in the numbers and it is desired to maintain realism in such test data, especially if the data involves very large numbers which must be maintained in sequence between test suite executions. The most expedient solution here is to create a testing database and to programmatically get the input data from these rows, perform any parsing required, use the data as needed in the Functional Tester scripts, possibly updating the data within Functional Tester from one script to another, and then update or insert rows into the testing database with the data that should be used in the next test suite execution. Refer to the Decorator design pattern in
Patterns: Elements of Reusable Object-Oriented Software.
You will need to install a database driver for the database that you will be using. For the testing framework described in this article, a SQL Server testing database was created. Download my code at bottom of the article for loading the Type 4 JDBC driver for SQL Server. Copy the three driver file jars to: C:\Program Files\IBM\Rational\SDP\6.0\eclipse\jre\lib\ext.
To enable tighter cohesion in your Functional Tester classes, you might want to isolate the code for actually interacting with your testing database and then just pass SQL statements to these methods as parameters (Download the code for interacting with the testing database at the bottom of the article).
If you want to afford your testing framework the maximum amount of abstraction of SQL-based database calls from your Functional Tester scripts, you can implement the design patterns, Mapper and Query Object, which allows you to make generic requests for information in your Functional Tester scripts, deferring the construction of database-specific SQL syntax to other classes.
Method three: Algorithmic generation of input data
Generating input data programmatically by algorithmically retrieving data from XML or relational databases is the core engine of a generic testing framework. This article presents a version of method three in which the test design pattern, Combinational Function Test (
Object-Oriented Systems Models, Patterns, and Tools
) is implemented. In this pattern, the actions resulting from each combination of a set of input values is evaluated.
One effective method for storing the possible input values for a set of GUI controls is to put the controls and their possible values into an XML file. With the current trend in application development leaning towards dynamic, execution-time determination of which controls to display, there is a good chance that the development team on your project will already have defined the controls and their possible values in an XML configuration file. But if they haven't, you can make such an XML file in a couple of minutes. Download sample XML description of GUI controls and their possible values at the bottom of the article). You will want to put in the values from the dropdown lists of any combo boxes. For text boxes, you can put in specific instances of categories of values that have business rule consequences (that is, put in a 1 and a 12, if the business rules state that values between 0 and 10 should cause one result, while values between 11 and 20 should cause another result).
It is possible to put in conditions instead of values (for example, "< 20). This would require some extra coding on your part to parse the conditions, which could become quite complex (refer to the Interpreter design pattern in
Design Patterns Elements of Reusable Object-Oriented Software
).
To access the XML file, we will use the Document Object Model (DOM). This method requires less programming than other methods and has the advantage of keeping all of your XML access code right inside a Functional Tester class. The usual disadvantage cited in using the DOM is that it requires a lot of RAM, but that should not be a problem with the small size of the XML files required to define the GUI controls for your application. It would allow for easier maintenance, if you were testing a constantly changing application, to create a separate XML document for each screen, or some other logical separation. The DOM parsing of an XML file that contains your GUI controls and the possible values for these controls is fairly straightforward, once you understand the document traversal methods. Basically, you create classes to set up the DOM to work with your XML document, filter the XML nodes, and process the results of that filter. Download the code at the bottom of the article to do all three of these activities.
As shown by these classes, the strategy employed in this article is to retrieve nodes by specifying the following:
- The root of the desired nodes
- The parent of the desired nodes
- The name of the desired nodes
The filter also specifies the number of siblings necessary to reach the comparison node. To facilitate making the testing framework generic, you should use variables for such values as the names of these filtering nodes, and set the values of these variables in a central location. This will make it easier to configure your testing framework for the next application under test. One value of using a contemporary tool like Functional Tester is that it makes it even possible to bring in powerful applications like the DOM to assist in complex tasks such as creating a generic testing framework.
Another value of using Functional Tester is that, by virtue of its support for Java, it provides robust support for Collections, which is absolutely essential in creating any kind of framework. In the testing framework being described here, the names and possible input values of all of the GUI controls that will be involved in the Combinational Function Test pattern implementation are stored in a Vector of Vectors. The Vector with index 0 in this super-Vector contains the names of each GUI control participating in the implementation. Each of the rest of the constituent Vectors contains the possible input values for a single participating GUI control. One task that must be accomplished in creating your framework is to construct a mapping to equate the three names that each GUI control will probably have: the XML file, the application under test code, and the Functional Tester Test Object Map. You can make a class that uses regular expressions for this, or if naming standards are strictly enforced in your organization, you could use a simple edit.
Now that we have all of the participating input values stored in Vectors, we can begin the task of generating all possible combinations of them. The method used here is to recursively form cross products of the values stored in each of the GUI-control-possible-value Vectors (we will call these Vectors, "Permutation Groups"). We start by forming an ArrayList containing a set of ArrayLists, in which each of the constituent ArrayLists contains exactly one element of the first Permutation Group. Then, a new ArrayList is created which contains the cross product of the contents of each of the constituent ArrayLists (of the super-ArrayList) with all of the elements in the second Permutation Group. With each recursive step, cross products with one additional Permutation Group are formed.
At the end of the process, we have a set of ArrayLists (test cases), each of which contains valid values for the participating set of GUI controls (Download the code at the bottom of the article.). The order of values in the ArrayLists corresponds with the order of GUI controls in the XML file (refer back to the code for the XML file configuration). Your framework should assign a number to each test case to allow for easy rerunning of failing test cases.
Reducing number of generated test cases
Of course, working with the output of the generation of all possible permutations of test inputs could quickly become unfeasible (depending on the application under test). For example, if you have ten participating GUI controls and if each of these controls can take just three possible values, you will have generated 59,049 test cases! There are several ways to limit this number. The easiest way is to put tags in the XML file that will cause only the critical GUI controls and values to be considered. Another way to allow your framework to generate all of the test cases, but then only process the first n, or process only randomly selected ones (parameters defining these selection criteria could be defined in a single, globally accessible class, to ease variable configuration for each application under test). Still other methods involve invoking business rules in the code that generates the test cases (that is, "if the value for GUI control A was less than 20, but the value for GUI control B was greater than 12, then discard this test case").
 |
Automatic production of expected results
Once the classes to auto-generate the test inputs have been created, you can create the Functional Tester test scripts that will use these inputs. The overall scheme here will be to follow a version of the test design pattern created by Dr. John D. McGregor: Sequence test cases using test scripts
and the test design pattern created by Robert V. Binder: Test Case/Test Suite Method. The overall structure of our testing framework will be to have a single controller test script which performs two functions: 1) cleans up after the previous test script and sets up subsequent test scripts, and 2) calls other modularized test scripts. The more cohesive these modular scripts are, the better, as this will promote script reusability. The Functional Tester/Java syntax for calling other scripts is: callScript ("OtherModularizedScriptName");. It is helpful to separate the callScript statements in the controller script with System.out.println statements that write comments onto the console indicating the sequence of events that occurred during processing of the controller script.
Following the assumption that there are a large number of test cases for the application under test, we recognize that recording each test case to create the modularized test scripts must be foregone in favor of a more generic solution. The Functional Tester tool allows us to implement the Prototype design pattern (Design Patterns Elements of Reusable Object-Oriented) to select the desired functional testing methods at execution time. This is accomplished by using Reflection to make parameterized calls to methods in the Functional Tester helper classes (Download the code at the bottom of the article). Functional Tester generates these helper classes automatically whenever a script is recorded.
The Functional Tester helper classes contain references to GUI objects in the application under test. The Functional Tester concept of adding GUI controls to a Test Object Map, which can be used by multiple test scripts, implements Fowler's Identity Map design pattern effectively. Functional Tester acquires the properties for these GUI objects through a two-step process performed by the tester:
- Use the Functional Tester GUI control recognition tool. That is, drag the "hand" icon (see Figure 3) over the control and release to enter the object into a Functional Tester Test Object Map.
- Add the Test Object Map to whichever test scripts need access to that control (from the Functional Test Perspective top menu bar: select Script > Open Test Object Map > select an object > Add to Script.
Figure 3. The Drag Hand Selection method
Using the process outlined above, you should be able to create modular, reusable scripts that will be called by the test script controller that you create. One thing that is missing, however, is a way to dynamically control your test suite. For this, your testing framework could use a GUI interface through which the test team can control runtime parameters (of course, Functional Tester also allows you to enter command line parameters when your test controller script starts). Fortunately, Functional Tester gives you access to any GUI that you want to create with the Standard Widget Toolkit (SWT).
First, you will need to install SWT into your Functional Tester (Eclipse) environment (refer to the book,
SWT A Developer's Notebook
for step-by-step instructions). Suppose that we want a simple dialog window in which to enter some runtime parameters. Then you need to make an SWT shell class and a class for the dialog window itself (Download the code at the bottom of the article. The finished dialog window at testing framework execution time is displayed in Figure 4.
Figure 4. Runtime display of SWT input dialog window
Automated Comparator
Now that your testing framework contains generic mechanisms for inputting test case data and for executing the test cases, you need a way to automatically verify results. There are several options available for comparing baseline data with actual results. Functional Tester provides three types of verifications: Static, Manual, and Dynamic (search for the topic, IftVerificationPoint, in the Functional Tester Help for complete descriptions). To create a generic testing framework, your best choice is probably Manual, because it allows for maximum programmatic control and efficient, high volume verification. Of course, you could also write your own code, but the Functional Tester methods already write results to the Playback log. So, your testing framework would contain scripting such as:
vpManual ("vpResultChecker1", strExpectedResult, strActualResult).performTest();
Then, you would read the expected results from either an XML file or a relational database table.
You would want to correlate each test case ID with an identified expected result. In large-volume testing environments, you might also want to create a GUI to display all verification failures (see Figure 5).
Figure 5. A sample GUI displaying verification discrepancies
Wrap-up
Now you have the core of a generic testing framework. Functional Tester has provided both its own direct support, as well as support for other auxiliary products through its open architecture. The examples in this article have been relatively complex, as compared with what you will have to do if you ever want to implement the other, simpler test design patterns that exist. Of course, as stated earlier, Functional Tester implements many of these patterns itself and allows for direct implementation of others rather simply. Additional coding to implement the remaining, simpler test design patterns would be orders of magnitude simpler than what was shown here, and thereby extremely easy to do in Functional Tester. Armed with this generic testing framework, you can vastly reduce testing time on your future testing projects.
Download | Description | Name | Size | Download method |
|---|
| All the code referenced in this article | All_code_for_RFT_article.zip | 42 KB | HTTP |
|---|
Resources
- For information on DOM visit the Object Model (DOM) Level 2 Core Specification Web site.
- Learn more about Dr. John D. McGregor's two test design patterns: an architecture for object-oriented component testing and test cases using test scripts.
- Download the free Type 4 JDBC driver for SQL Server.
- Read more about object-oriented systems in
Testing Object-Oriented Systems: Models, Patterns, and Tools
by Robert V. Binder (Addison-Wesley Object Technology Series).
- Read more about design patterns in
Design Patterns: Elements of Reusable Object-Oriented Software
by Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides (Addison-Wesley Professional Computing Series).
- Read more about SWT in
SWT: A Developer's Notebook
by Tim Hatton (O'Reilly).
- Check out the Software Testing Patterns Web site.
- Find more patterns in the Catalog of Patterns of Enterprise Application Software.
- Read more about test-driven development in
Test-Driven Development : By Example
by Kent Beck (Addison-Wesley Signature Series)
- Share your questions and views on this article with the author and
other readers in the Rational discussion forums.
- To learn more, visit the IBM Rational Functional Tester product area on developerWorks Rational. You'll find technical documentation, how-to articles,
education, downloads, product information, and more.
- Get involved in the developerWorks community by participating in
developerWorks
blogs.
-
Browse for books on these and other technical topics.
About the author | | | Randy Masciana is a testing consultant currently engaged with the Bureau of Laboratories at the Florida State Department of Health under a U.S. Government grant from the Centers for Disease Control. Randy is also president of three organizations: Big Iron Software, Inc., Non-Figurative Analysis, and Hyperbolic Games. He specializes in analysis, as well as the creation of custom software solutions (in English, UML, VBA, VB.NET, C#.NET, Java, and Cobol) for complex problems across a number of industrial and government domains. He can be reached by e-mail. |
Rate this page
| |
