 | Level: Advanced Michael Kelly (Mike@MichaelDKelly.com), Senior consultant, Fusion Alliance Allen Stoker (Allen.Stoker@PortalWizard.com), Technologist, RAM Division, Liberty Mutual
06 Sep 2005 Updated 10 Nov 2005 This article is for developers or testers interested in testing a database directly from their IBM Rational Functional Tester scripts. The information is explained using examples and detailed instructions.
Editor's Note: This article is intended for readers who are comfortable coding. For readers who would like to avoid coding and use an utility to verify database data, see the "Verify your order in a database" section of the tutorial Test automation scripting: Useful coding techniques for IBM Rational Functional Tester scripts.
When performing database testing, you are concerned with verifying that your data is being stored by the system in the manner in which you expect, and in such a way that no errors are introduced to the data. This article will present some scenarios along with examples of database tests that might uncover design flaws that may result in data corruption, unauthorized data access, lack of data integrity across multiple tables, and lack of adequate transaction performance.
This article is intended for developers and testers who:
- Are familiar with the basic SQL commands (we won’t spend time explaining the SQL queries used in this article)
- May have done some database testing manually
- Have an interest in automating some of that testing
This article will first walk through setting up IBM® Rational® Functional Tester (RFT) to connect to a Java™ Database Connectivity (JDBC™) data source. Next you will walk through a sample application (complete with database defects), and then look at example tests you might automate for database testing.
Note: This article was written using RFT 6.1, Apache Tomcat 5.5.9, Java™ 2 Platform, Standard Edition (J2SE®) 5.0, Java™ 2 Runtime Environment, Standard Edition (JRE™) 5.0, and Hypersonic SQL (HSQLDB) 1.7.1.
Introduction to database testing
There are several advantages to automated database testing. First, automated database testing is faster and more reliable the traditional GUI-based test automation. It’s faster because you don’t have to process the GUI to do your testing. That is, starting the application, logging in, navigating to the feature you want to test, and running the test is significantly slower then just creating a connection to a database and executing a query. In addition, if you use a tool like RFT, which uses a multithreaded language, you can execute multiple tests in parallel. For the same reasons that your tests are faster, they are more reliable. You no longer need to update your scripts for GUI-level changes.
Another advantage to testing the database directly is that there are no GUI-level blocking bugs. A blocking bug is a bug that prevents you from executing your other tests. For example, if you were testing a shopping cart and wanted to test credit card processing, but you couldn’t add any items to your shopping cart because the Add to Cart functionality was broken, then your testing would be blocked by another bug. If you test directly at the database, you are not immune to blocking bugs, but they are typically much less frequent.
Database testing is as simple or as complex as you want it to be. As you start to dive into your database, you may find it helpful to identify:
- The types of database activity (insert, delete, update) that will be invoked
- When these transactions will occur within the application-under-test
- The calculations or processing rules used by the application
- Any time-critical transactions, functions, conditions, and possible causes of poor performance
Not all database tests need to be difficult. There are also some simple tests you can perform, such as checking:
- Data fields to ensure proper data types
- For proper spacing and length
- Data formats that may not be enforced by a data type
- For duplicate data
- Reference data to ensure that the data you are expecting is there
Note: It is a good practice to keep backup copies of test files and test databases. Before reporting an error, check your working copies of the input and comparison files against the backups.
Setting up RFT
In his article
Establishing a database connection in IBM Rational Functional Tester scripts
, Fariz Saracevic walks you through how to establish a JDBC connection. Saracevic provides a step-by-step process for getting RFT set up, and we don’t want to reproduce those steps here. Instead, we will simply provide updates, screenshots, and source listings that reflect the differences in JDBC connections.
In his article, Saracevic establishes a connection to an Oracle database. This example application uses HSQLDB. If you download the sample application and extract the files, the HSQLDB JDBC driver can be found in the top-level extraction directory. The following are updated figures for the steps in Saracevic’s article, based on the new data source.
Figure 1 shows the properties screen after you add the driver.
Figure 1. Adding the driver to your project
Figure 2 shows the data used to create the new JDBC connection class.
Figure 2. Specifying the name for your new connection class
Listing 1 shows the code for the JDBC connection class. The code has been updated for the sample application data source.
Listing 1. The JDBC connection class
package Scripts;
import java.sql.Connection;
import java.sql.Driver;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
public class HSQLConnection {
public static void main(String[] args) throws SQLException {
new HSQLConnection();
}
HSQLConnection() throws SQLException {
connect();
}
Connection connection;
void connect() throws SQLException {
Driver driver = new org.hsqldb.jdbcDriver();
DriverManager.registerDriver(driver);
connection = DriverManager.getConnection(
"jdbc:hsqldb:C:/LibraryServer/data/library", "sa", "");
//The location "C:/LibraryServer/data/library" is relative
//to where you unzip the Library Program
connection.setAutoCommit(true);
}
public ResultSet returnQuery(String arg) throws SQLException {
ResultSet resultset = null;
Statement statement = null;
try {
statement = connection.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
resultset = statement.executeQuery(arg);
}
catch (Exception e)
{
e.printStackTrace();
}
return resultset;
} // returnQuery
public int executeUpdate(String arg) throws SQLException {
int result = 0;
Statement statement = null;
try {
statement = connection.createStatement();
result = statement.executeUpdate(arg);
}
catch (Exception e)
{
e.printStackTrace();
}
return result;
} // executeUpdate
}
|
The code in Listing 2 is a simple test script that you can run to test your connection with the sample application database.
Listing 2. Code to test the JDBC connection
package Scripts;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import resources.Scripts.TestConnectionToDBHelper;
import com.rational.test.ft.*;
import com.rational.test.ft.object.interfaces.*;
import com.rational.test.ft.script.*;
import com.rational.test.ft.value.*;
import com.rational.test.ft.vp.*;
import resources.Scripts.TestConnectionToDBHelper;
import resources.Scripts.TestConnectionToDBHelper;
public class TestConnectionToDB extends TestConnectionToDBHelper
{
public void testMain(Object[] args)
{
HSQLConnection connection;
ResultSet results;
try {
connection = new HSQLConnection();
results = connection.returnQuery("Select * from BOOK");
ResultSetMetaData rsmd = results.getMetaData();
int columns = rsmd.getColumnCount();
while (results.next()) {
for (int i = 0; i < columns; i++) {
System.out.print(results.getObject(i + 1) + " ");
}
System.out.println();
}
}
catch (Exception e) {
e.printStackTrace();
}
}
}
|
Setting up the sample application
Installing the application
If you wish to use the example application, you will first need to download the application. This application requires the Tomcat server, which is freely available from the Apache Software Group, and the Sun™ J2SE Developer Kit (JDK™) can be downloaded from Sun’s web site (for both addresses, see the
Resources
section). The application is known to work with either of the following configurations:
- Tomcat 5.0.28 using JDK 1.4.2_06
- Tomcat 5.5.9 using JDK 1.5.0_04
You must first download the installation files. The installation files are in a zip format, so you will need to use a zip-compatible program to extract the files into a local folder. Once extracted, you should see a folder structure similar to that in Figure 3.
Figure 3. Extracted project files
Using the installation script
The CreateLibrary.bat file will set up the application from a DOS (or CMD) prompt. To do this, follow these steps.
- Start a command prompt. You can use the Windows start menu, select Run, and enter
cmd.
- Change the current folder to the location where you extracted the installation files.
- The installation script requires two environment variables to be set.
JAVA_HOME must be set to the location of the JDK and CATALINA_HOME must be set to the location of the Tomcat server.
- Once these variables are set, just run the
CreateLibrary batch file with a single parameter indicating where you want the server instance created. The window in Figure 4 shows the commands executed to create the instance. You’ll need to use values appropriate for your personal machine setup.
Figure 4. Installation script command execution
- Once the server instance is created, you can start the server by changing to the server instance folder and running the
server.bat file, as shown in Figure 5.
- To stop the server, simply press Ctrl-C.
Figure 5. Running server.bat
Since individual machines can have vastly different configurations, there is no promise that the installation script will work for you. If it seems that you have a personal configuration significantly different from the example, you should review the Tomcat administration documentation located on the Apache web site. All of the files required to install the application manually in a Tomcat server are readily available in the installation file download. The details of configuring and administering a Tomcat server are beyond the scope of this document.
A quick overview of the application
Once you’ve started the server you can open a web browser. The URL will be based on the Tomcat configuration, but will typically be that in Figure 6: http://localhost:8080.
Figure 6. Launching the Personal Library program
This application was built with a unique feature, the ability to break itself! There are multiple bugs that are enabled when you click the break it link on the About page. This option can be switched back and forth and is retained in memory until the server is stopped. If you restart the server, the application will default back to working mode.
There are only a few basic functions to this program. It provides the ability to manage tables of books, publishers, and authors. The menu links all appear on the left side of the application. The database installed with the application already contains some basic data. You’ll start by looking at the Search page shown in Figure 7.
Figure 7. Searching for a book
You can enter some or all of a book title and search by clicking the Search button. The page will re-display with all matches. The search uses both pre- and post- wildcards, and is case sensitive. Searching for k should find any book with a lower case k in the title. Each book displays with the name of the author and publisher. This data is retrieved from two reference tables.
You can also change a book entry by clicking the title, which will take you to the Book Edit page shown in Figure 8. If you click the Add a Book option from the menu, you will arrive at the same screen, but in Add mode.
Figure 8. Edit book information
While working on the Edit Book page, you can select both the author and the publisher from the lists to the right. All reference values appear in the respective list. Conversely, if you manually type in a name for either the author or the publisher, the entered data will be added to the appropriate reference table.
The reference lists for authors and publishers can be accessed from the menu on the left. Both of the reference data lists work in a similar manner. As shown in Figure 9, you’ll select the Authors List for this example.
Figure 9. Author list reference data
Selecting an author will allow you to edit an existing author name, as shown in Figure 10. There is a link at the bottom of the page to open the edit page in Add mode.
Figure 10. Changing author information
This application was written using the simplest possible approach. It should be emphasized that this code should not be used as an example to construct an actual production application.
Examples of database testing using RFT
Now that you have the RFT configured and Personal Library installed, you can write some tests. The types of database tests contained in this section are not meant to be exhaustive. They are only meant to provide you with ideas and examples of how you might test your database applications. Some of them will make sense for your application and context, and some of them won’t.
For example, we really wanted to show an example of a database performance problem. However, Allen couldn’t write a query slow enough to make a noticeable difference given the simplicity of our application and the relatively small set of data we have available. We recommend an exploratory approach as you develop your database tests. Write some initial tests, learn from them, then either continue down that path (if it’s fruitful) or look at other types of tests or testing.
Let’s get started with the examples. Click the About the Library link in the Personal Library program and ensure that the application is broken.
Data correctness
Your first test will look at data correctness. When testing for data correctness, you are looking for things like data types, the truncation of data, encoding differences between the GUI and the database, and other small changes that may affect your data when it is written to the database.
The error
- Click Add a Book.
- Enter a Title, Author, Publisher, and Year.
- For Price, enter the value $27.95.
Your screen should look like that shown in Figure 11.
Figure 11. Values for adding a book
- Click Save.
- Search for the title of the book you just entered and open the results.
- If you look at the price that was persisted (Figure 12), you can see that it was saved as an integer, but the screen let you enter a float.
Figure 12. Type conversion of Price when saved
The test
Your test script will involve the steps listed previously, with the exception of the last two where you verify that the values were saved correctly.
- Record steps 1 – 4 in a new script using RFT.
- Before you stop recording, close the Personal Library program.
- Insert the code shown at the end of Listing 3 to validate that the data was saved correctly.
Listing 3. Code to test for type conversion
package Scripts;
import java.sql.ResultSet;
import resources.Scripts.DataCorrectnessHelper;
import com.rational.test.ft.*;
import com.rational.test.ft.object.interfaces.*;
import com.rational.test.ft.script.*;
import com.rational.test.ft.value.*;
import com.rational.test.ft.vp.*;
public class DataCorrectness extends DataCorrectnessHelper
{
public void testMain(Object[] args)
{
// Start the application and click Add a Book
startApp("http://localhost:8080/");
link__AddABook().click();
// Enter book data
text_title().click(atPoint(21,6));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputChars(
"How to Break Software Security");
list_authorList().click();
list_authorList().click(atText("James A. Whittaker"));
list_publisherList().click();
list_publisherList().click(atText("Addison-Wesley"));
text_year().click(atPoint(4,10));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputKeys(
"{ExtDelete}{ExtDelete}2004");
text_price().click(atPoint(4,10));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputKeys(
"{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}27.95");
button_savesubmit().click();
browser_htmlBrowser(document_library(),MAY_EXIT).close();
// Validate the data in the database
HSQLConnection connection;
ResultSet results;
try {
connection = new HSQLConnection();
results = connection.returnQuery(
"Select price from BOOK where title = 'How to Break Software Security'");
results.next();
if(results.getString(1).equals("27.95")) {
logTestResult("Database check for book price matched.", true);
} else {
logTestResult("Database check for book price failed.", false,
"Expected '27.95' but found '"+results.getString(1)+"'");
}
connection.connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
|
The results
If you play the script back on the broken version of the Personal Library application, you will see the information in Figure 13 in your log entry.
Figure 13. Failed database check log entry
If you run the script on the working version and delete the record you just added, the script passes.
Data location
The next test will check to ensure that your data is saved where you expect it to be saved. For this test, you will update the author of a book and make sure that the author information is written to the correct place.
The error
- Click Add a Book.
- Enter the book information and click Save.
- Search for the title of the book you just entered and open the results.
- Notice that all the information you entered (except possibly the price – but we already know about that bug) is correct. Specifically, notice the author and publisher are correct.
- Without changing any of the data, click Save.
- Again, search for the title of the book and open the results.
- This time you should notice one of the two following issues:
- The Publisher field contains the incorrect publisher
- The Publisher field contains the value "Read Error!"
- The error is that on an update (this is not a problem when you first enter the book) the publisher ID in the database is saved with the author ID. That means if you have more authors then publishers you will get the "Read Error!" and if you have more publishers then authors you will most likely display the incorrect publisher. It’s most likely because there is always the chance that your publisher and author have the same ID in the database.
The test
The test script is exactly like the first test script, only this time you will update the record after you add it, and change your SQL query to check for the author ID and the publisher ID on update.
- Record steps 1 – 5 (previously) in a new script using RFT.
- Before you stop recording, close the Personal Library program.
- For this test you will need two separate SQL queries. The first is to determine the author ID and publisher ID before we update, and the second is to determine them after we update. Both queries are shown in Listing 4.
Listing 4. Code to test data location
package Scripts;
import java.sql.ResultSet;
import resources.Scripts.DataLocationHelper;
import com.rational.test.ft.*;
import com.rational.test.ft.object.interfaces.*;
import com.rational.test.ft.script.*;
import com.rational.test.ft.value.*;
import com.rational.test.ft.vp.*;
public class DataLocation extends DataLocationHelper
{
public void testMain(Object[] args)
{
// Start the application and click Add a Book
startApp("http://localhost:8080/");
link__AddABook().click();
// Enter book data
text_title().click(atPoint(21,6));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputChars(
"How to Break Software Security");
list_authorList().click();
list_authorList().click(atText("James A. Whittaker"));
list_publisherList().click();
list_publisherList().click(atText("Addison-Wesley"));
text_year().click(atPoint(4,10));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputKeys(
"{ExtDelete}{ExtDelete}2004");
text_price().click(atPoint(4,10));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputKeys(
"{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}{ExtDelete}27.95");
button_savesubmit().click();
// Get the publisher ID and author ID after you save it the first time
HSQLConnection connection;
ResultSet results;
String authorID = "", publisherID = "";
try {
connection = new HSQLConnection();
results = connection.returnQuery(
"Select authorid, publisherid from BOOK where title = 'How to Break Software Security'");
results.next();
authorID = results.getString(1);
publisherID = results.getString(2);
connection.connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
// Search for the book and Save it again
text_title2().click(atPoint(37,11));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputKeys(
"How to Break Software Security");
button_searchsubmit().click();
link_howToBreakSoftwareSecurit().click();
button_savesubmit2().click();
browser_htmlBrowser(document_library(),MAY_EXIT).close();
// Validate the data in the database matches what is was before the update
try {
connection = new HSQLConnection();
results = connection.returnQuery(
"Select authorid, publisherid from BOOK where title = 'How to Break Software Security'");
results.next();
if(authorID.equals(results.getString(1))) {
logTestResult("Author ID matched.", true);
} else {
logTestResult("Author ID failed.", false,
"Expected '" + authorID + "' but found '"+results.getString(1)+"'");
}
if(publisherID.equals(results.getString(2))) {
logTestResult("Publisher ID matched.", true);
} else {
logTestResult("Publisher ID failed.", false,
"Expected '" + publisherID + "' but found '"+results.getString(2)+"'");
}
connection.connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
|
The results
If you play the script back on the broken version of the Personal Library application, you will see the information in Figure 14 in your log entry:
Figure 14. Failed database check log entry
If you run the script on the working version and delete the record you just added, the script passes.
Data persistence
Now let’s try something different. In this test, you will look at data persistence. If you wanted to test your Save, Delete, and Cancel buttons on the GUI, you would have to do a lot of opening, closing, and re-opening of screens and data. By connecting to the database, you can get around the need to reopen data. For example, there is a bug in the Personal Library for publishers. If you click Cancel, the application actually saves your changes.
The error
- Click Publisher List.
- Click on Apress.
- Change the name to Apress2.
- Click Cancel.
- Notice your change was saved.
The test
- Record steps 1 – 4 in a new script using RFT.
- Before you stop recording, close the Personal Library program.
- Insert the code shown at the end of Listing to validate that the data was saved correctly.
Listing 5. Code to test for data persistence
package Scripts;
import java.sql.ResultSet;
import resources.Scripts.DataPersistenceHelper;
import com.rational.test.ft.*;
import com.rational.test.ft.object.interfaces.*;
import com.rational.test.ft.script.*;
import com.rational.test.ft.value.*;
import com.rational.test.ft.vp.*;
public class DataPersistence extends DataPersistenceHelper
{
public void testMain(Object[] args)
{
// Start the application and click Publisher List
startApp("http://localhost:8080/");
link_publishersList().click();
// Change and Cancel
link_apress().click();
text_publisherName().click(atPoint(65,5));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputChars("2");
button_cancelsubmit().click();
browser_htmlBrowser(document_library(),MAY_EXIT).close();
// Validate the data in the database
HSQLConnection connection;
ResultSet results;
try {
connection = new HSQLConnection();
results = connection.returnQuery(
"Select name from PUBLISHER where publisherid = '14'");
results.next();
if(results.getString(1).equals("Apress")) {
logTestResult("Cancel button worked correctly.", true);
} else {
logTestResult("Cancel button failed.", false,
"Expected 'Apress' but found '"+results.getString(1)+"'");
}
connection.connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
}
}
|
The results
If you play the script back on the broken version of the Personal Library application, you will see the information in Figure 15 in your log entry:
Figure 15. Failed database check log entry
If you run the script on the working version and correct the name of the publisher you just changed, the script passes.
Fault injection: Inserting values into the database to ensure proper error handling
The final test is a fun one. While this may not be the best type of test to automate, we wanted to include it as an example of what you can do within your scripts. In this test, you are going to insert valid database values into your database that the GUI developers didn’t expect. When we say they didn’t expect it, we mean they didn’t add GUI-level validation for the value after they read it from the database. This is a common problem causing errors in later software releases, where new interfaces (new ways of saving data) have been introduced that the original developers didn’t think of.
The error
- Click Add a Book.
- Enter the book information, but attempt to leave the price blank (not a zero, but null).
- Click Save.
- You should get a prompt asking you to enter a price. The developer is forcing you to enter a price. This is your best clue that the developer did not equip the application to handle a null value (or whatever value they are preventing).
The test
- In Listing 6, we start the script by inserting a book into the database where the book price is null. Create a new RFT script and insert that code at the start of the script.
- Record the following steps in your script to verify the error:
- Click Search for a Book.
- Enter "Robust Java".
- Click Search.
- Click Robust Java: Exception Handling, Testing and Debugging.
- Verify the
NullPointerException and log an error.
Listing 6. Code to inject a value into the database
package Scripts;
import java.sql.ResultSet;
import resources.Scripts.FaultInjectionHelper;
import com.rational.test.ft.*;
import com.rational.test.ft.object.interfaces.*;
import com.rational.test.ft.script.*;
import com.rational.test.ft.value.*;
import com.rational.test.ft.vp.*;
public class FaultInjection extends FaultInjectionHelper
{
public void testMain(Object[] args)
{
// Insert the fault into the database
HSQLConnection connection;
int result;
try {
connection = new HSQLConnection();
result = connection.executeUpdate(
"Update BOOK Set price = null Where title = 'Robust Java: Exception Handling, Testing and Debugging'");
connection.connection.close();
}
catch (Exception e) {
e.printStackTrace();
}
// Start the application and Search
startApp("http://localhost:8080/");
link_searchForABook().click();
text_title().click(atPoint(31,13));
browser_htmlBrowser(document_library(),DEFAULT_FLAGS).inputChars(
"Robust Java");
button_searchsubmit().click();
link_robustJavaExceptionHandli().click();
// Verify error
if(ErrorReport_textVP().performTest(2.0, 20.0)) {
logTestResult("Found java.lang.NullPointerException.", false);
}
browser_htmlBrowser(document_library(),MAY_EXIT).close();
}
}
|
The results
If you play the script back on the broken version of the Personal Library application, you will see the information shown in Figure 16 in java exception and that shown in Figure 17 in your log entry:
Figure 16. Application JavaException
Figure 17. Fault injection log results
If you run the script on the working version, the script passes.
Next steps
As mentioned previously, we wanted to add an example of a performance test, but it just didn’t make sense given the application. In addition, we also wanted to show an example of a test for timing. Many times applications save data at specific times. For example, if your application has a wizard, sometimes data is saved every time you click Next, and sometimes it’s not saved unless you explicitly click Save. Database timing and performance tests are important, and they can often save time debugging more complicated tests that take place at the system level. Be sure you are running these tests if they make sense for your application.
You can also use your newly found power to test the reference data in your application. If you know what data your application should have "out of the box," you can run a quick query (perhaps in your smoke test) to ensure that all the data you are expecting is there. For example, in your Personal Library program you could run a test to ensure that all of your publishers are in the database before we start any other testing.
Now that you’ve run through some of these examples, write some simple tests against your application. Try to implement each example in this article using your application. Also try a performance-related test and a test dealing with timing. As you work through some simple tests with your application, we recommend
Using JDBC to create database objects
by James W. Cooper as a good reference. The more tests you write, the easier it gets.
Download | Description | Name | Size | Download method |
|---|
| Code sample files. | library_application.zip | 575 kb | HTTP |
|---|
Resources Learn
Get products and technologies
Discuss
About the authors  | |  | Michael Kelly is currently a senior consultant for Fusion Alliance in Indianapolis. He's had experience managing a software automation testing team and has been working with the Rational tools since 1999. His primary areas of interest are software development life cycles, software test automation, and project management. Mike can be reached by e-mail. |
 | |  | Allen Stoker is a technologist with Liberty Mutual in the Regional Agency Markets (RAM) division. He's been implementing systems on a variety of platforms (including IBM mainframe, Windows, and J2EE) since 1983, and today focuses primarily on quality-driven development in the J2EE environment. He was recently responsible for driving the implementation of unit-testing practices across the RAM organization. Allen can be reached by e-mail. |
Rate this page
|  |