Level: Introductory Keyton Weissinger (keyton@weissinger.org), Principal, Key Focus Inc.
01 Jun 2002 Puffin is an open source framework for testing Web applications. With
Puffin, you can build dynamically driven regression tests for even the most complex
Web applications. Written in Python, Puffin is easily extended to handle even obscure testing conditions.
Introduction
As important as unit testing is, it fails to test the overall functionality
of an application as a system. Just because each functional component
works as designed through the construction of its unit tests, that doesn't
mean that those components — when brought together into a coherent
application — will work together to meet the system's overall functional
requirements. System-wide functional testing is an important part of
deployment of all applications, and Web applications are no exception.
However, the tools to enable full system testing are either
cost-prohibitive or overly complex — often to the point that system-wide
testing is done only cursorily or a human being is forced to test each functional area manually.
Human testing can never (and arguably should never) be completely
removed from the testing process. However, it can be argued that improving
the automation of testing can free the quality engineer to focus on the most
complex functionality.
Complex testing aside, there is another benefit to creating automated
smoke tests of your entire Web application. Most
systems — especially large ones — are in a constant state of change in
response to evolving business needs.
Just as unit testing allows you to make pervasive (even extreme) changes
to a given unit with confidence that those changes
will not affect that unit's behaviour, a smoke test of the entire system
would allow you that same confidence across
the entire system.
Finally, system-wide smoke tests — specifically for Web applications —
solve a problem of many unit-testing frameworks: changes in technology. Unit tests — even a long series
of unit tests — are very
technology-specific, forcing you to rewrite every component (and its
corresponding unit test) at the same
time or testing only some portion of the units making up the system. For
example, suppose you are converting
a CGI application to a JSP/servlet architecture. In this scenario, there are
doubtless some components that
can be converted separately from the others. Only with a system-wide smoke
test will you be able
to test both the older CGI components and the newer JSP/servlet components
and their integration into a whole system during the migration
process.
This last capability points out a shortcoming in many system-testing tools:
technology specificity.
There are plenty of system-testing tools out there. Some are even open
source. However, most are technology-specific, making changes to the underlying technology difficult. The
result? Most people either write unmaintainable scripts to test the
back-and-forth HTTP traffic from the Web application, or worse, rely 100 percent on
human testing.
What's needed is a Web application-testing tool that allows for:
- End-to-end system testing with dynamically created test actions based
on results from previous test actions.
- Using the same test to test the same functionality — regardless of the
Web application's underlying technology. You should be able to test the Web application as if it were a proverbial
black box.
Furthermore, the following would be nice to have in a Web application-testing tool:
- Ease of use (less need for scripting, etc.)
- Extensibility
- Flexibility, flexibility, flexibility
This article introduces Puffin, a Web application-testing framework that
provides developers of even the
most complex Web applications all of the above features. This introduction covers how to
get and install Puffin, how to configure it to your
Web application's needs, and
how to write simple test plans.
Installing Puffin
Fortunately, installing Puffin is fairly simple (see Resources):
- Download and install the appropriate Python distribution for your
platform.
- Download and install the PyXML package for your platform.
- Download and install the eGenix base package of Python extensions.
- Download and uncompress Puffin.
In the puffin/doc folder are two documents: GettingStarted.htm and
UserGuide.htm. Read through the "Getting Started" document to see how to
set up the test application (used as our demo below). Tackle the "User Guide"
after you finish this article.
From this point on, we will assume that you have Puffin installed,
but have not necessarily run anything yet.
Framework configuration
Before you can start writing tests, you must configure the Puffin framework
to your Web application. You do this through a single XML file called (by
default, this can be overridden) puffinConfig.xml. In this file, we will
set up the following aspects of the Puffin system:
- Address of our Web application's server
- Logging channels for the framework itself and results processing
- Level of detail desired for the results reporting
- Universal response analyzers for checking the validity of all responses
in your Web application
- Possible test actions callable against the Web application
For this example, we will create a config file from scratch, though you
will likely just customize the one that comes with Puffin. For right now,
rename the puffinConfig.xml file that comes with Puffin to
puffinConfigORIG.xml or similar and create a new text file called
puffinConfig.xml. Add the following XML to it, and save it in the same
folder where puffin.py is located.
Listing 1. Create config file
<puffin>
<system/>
<testActions/>
</puffin>
|
As you can see, the config file has two main sections. The first
describes the various properties of the Web application system itself. The
second describes the possible test actions you can call in the test plans
you will create.
The first system element we need to specify is the server information.
This is as simple as it sounds. Just add a <server> element with the
location of your Web application server:
Listing 2. Add <server> element
<puffin>
<system>
<server host="www.mydomain.com" port="80"/>
</system>
<testActions/>
</puffin>
|
The <server> element has two attributes:
- The
host attribute specifies a host name.
- The
port attribute specifies a port number.
Note that we will assume you set up the demo application locally and
will, thus, set the server element accordingly (though this isn't strictly
necessary as localhost and port 80 are the defaults if no
<server> element is in place).
Listing 3. Set server element
<puffin>
<system>
<server host="localhost" port="80"/>
</system>
<testActions/>
</puffin>
|
Puffin knows where to find your Web application's server. Now we need to
set up how Puffin will tell you what is happening with your test plans as
they execute. There are three types of information that Puffin will attempt
to relay to you:
- A log of the activity of the Puffin framework itself
- A report describing the results of a given test plan's execution
- A mini-report of a failed task occurring during the execution of a
test plan
Puffin uses (and, dare I say, "abuses") a logging system based on the
popular log4j design by Vinay Sajip (see Resources) to present this information.
If you are familiar with log4j or log4py or pretty much any log4* product,
you will already understand logging priorities and logging
handlers and can skip the next two paragraphs.
The logging system used in Puffin allows you to set a priority threshold
for each channel of logging. So for example, if I set the priority of the
logging channel that handles messages from the framework itself to a level
of WARN, then only messages sent with WARN or above will actually get
logged. All others will be ignored. The priority levels are DEBUG, INFO,
WARN, ERROR, and FATAL.
Handlers are used to direct logging statements to targets. You may have
a handler that will direct your logging statements to the command line, to an e-mail
address, or (on Windows® NT) to the event log. You can stack
handlers, too. For example, I could set it up so that all failed task
logging messages for Puffin go to an e-mail address, and all framework
logging messages go to the console and to a file.
As you might guess, you configure the Puffin logging channels through
the exact same config XML file with which we have been working. Following are
three more entries.
Listing 4. Configure Puffin logging channels
<puffin>
<system>
<server>localhost</server>
<frameworkLogging channelPriority="WARN">
<handler type="StreamHandler">
<param name="msgFormat"><![CDATA[%(asctime)s %(name)s %(message)s]]></param>
</handler>
</frameworkLogging>
<reportLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinResults.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</reportLogging>
<failureAlertLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"<![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinFailures.log</param>
<param name="mode" eval="0"><![CDATA[a+]]>
</param>
</handler>
</failureAlertLogging>
</system>
<testActions/>
</puffin>
|
The three logging entries allow you to set the priority and set one or
more handlers. The above sets up the StreamHandler (console logging) for
the framework itself and file handlers for the results reporting and
the failed tasks alerts. See the Puffin User Guide for handlers
supported. The key attribute is the priority. All report and failure
information is logged at a level of WARN. If you do not set these
priorities to at least this level, no reports or failure alerts will be
written. The framework itself does some logging at WARN (mostly just
letting you know status) and some at higher (ERROR or FATAL) levels. It
also has lots of debugging in place as we are still only at a 0.8
release.
Puffin now knows where your server is and how to communicate with you. Now
we need to express how much information for test plan execution results you
want Puffin to communicate. We do this with a single additional line in the
config file.
Listing 5. How much information for test plan execution results
<puffin>
<system>
<server>localhost</server>
<frameworkLogging channelPriority="WARN">
<handler type="StreamHandler">
<param name="msgFormat"><![CDATA[%(asctime)s %(name)s%(message)s]]></param>
</handler>
</frameworkLogging>
<reportLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinResults.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</reportLogging>
<failureAlertLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinFailures.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</failureAlertLogging>
<resultsProcessor reportDetail="ALL"/>
</system>
<testActions/>
</puffin> |
For now, we'll leave the detail level at "ALL," which means that you will
get back more than you probably want for results of a test plan's
execution. But that's OK for learning the system. See the User Guide if
you want to trim it back a bit.
Before we complete our configuration (we only have one more entry for
now), we need to take a break to discuss how Puffin works in testing your
Web application.
Response analysis
As we will see, Puffin test plans are broken up into
a series of test actions. Each test action represents a single
request-response interaction with your Web application. We'll get into the
details of this interaction in a few moments. For now, we will discuss only
the response that results from the execution of a test action. Puffin will
analyze this response for criteria you specify and will use the results of
that analysis to deem a given test action a success or failure.
For example, if you are running a JSP-based Web application and have set
up an error page that says something like "You have experienced an error.
Please call support at ...," then Puffin can analyze the HTTP response
resulting from a test action and "look" for the phrase "experienced an
error." If Puffin finds that phrase upon executing a test action, that test
action will be considered a failure. As another example, you will want to
simply ensure that a given test action (Web application call) takes no more
than a set amount of time or that it always returns the HTTP status of
"200," meaning everything went OK.
Puffin performs these types of analyses using a ResponseAnalyzer object,
of which there are several types core to the system (and, of course, you
can extend Puffin with your own). You set these up using the config file
just like everything else. You can set up universal response analyzers that
Puffin will execute against every single test action's response or just for
a specific test action. The former type is set up in the <system>
element of the config file; the latter is set up with the individual test action.
We will cover test action-specific response analyzers in a later
article. Here is how to set up the universal ones (note that the top half
of the config file was snipped off for brevity).
Listing 6. Setting up a response analyzer
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</failureAlertLogging>
<resultsProcessor reportDetail="ALL"/>
<defaultResponseAnalyzerList>
<responseAnalyzer src="puffin" type="TimerResponseAnalyzer">
<param name="timeLimitSecs">10</param>
</responseAnalyzer>
<responseAnalyzer src="puffin" type="StatusResponseAnalyzer">
<param name="httpStatus">200</param>
<param name="httpStatus">302</param>
</responseAnalyzer>
</defaultResponseAnalyzerList>
</system>
<testActions/>
</puffin> |
As you can probably surmise, the above sets up Puffin so that any test
action that does not have its own response analyzers will have its response
analyzed using these two response analyzers:
-
TimerResponseAnalyzer checks to see that the response came back
in less than (in this case) 10 seconds.
-
StatusResponseAnalyzer checks to see that the server returns
either a "200" (for OK) or a "302" (for redirects) as an HTTP status code.
Now when you run a test plan, the response of every test action (without
its own ResponseAnalyzer) will be checked to see if it came back in the
time allowed and that the HTTP status code was 200 or 302. If either of
these results in a failure, Puffin considers that whole test action a
failure.
There are some other things you can set up as system-wide (flag settings
for security being on or off, whether or not to always send a cookie, etc.),
but we will leave it here for now. Check the User Guide for detail
on these more advanced features. Now let's dive into the test action
configuration. At this point, your puffinConfig.xml file should look like
this.
Listing 7. puffinConfig.xml file
<puffin>
<system>
<server>localhost</server>
<frameworkLogging channelPriority="WARN">
<handler type="StreamHandler">
<param name="msgFormat"><![CDATA[%(asctime)s %(name)s %(message)s]]>
</param>
</handler>
</frameworkLogging>
<reportLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinResults.log</param>
<param name="mode" eval="0"><![CDATA[a+]]></param>
</handler>
</reportLogging>
<failureAlertLogging channelPriority="WARN">
<handler type="FileHandler">
<param name="msgFormat" eval="0"><![CDATA[%(message)s]]></param>
<param name="fileName" eval="0">puffinFailures.log</param>
<param name="mode" eval="0"><![CDATA[a+]]>
</param>
</handler>
</failureAlertLogging>
<resultsProcessor reportDetail="ALL"/>
<defaultResponseAnalyzerList>
<responseAnalyzer src="puffin" type="TimerResponseAnalyzer">
<param name="timeLimitSecs">10</param>
</responseAnalyzer>
<responseAnalyzer src="puffin" type="StatusResponseAnalyzer">
<param name="httpStatus">200</param>
<param name="httpStatus">302</param>
</responseAnalyzer>
</defaultResponseAnalyzerList>
</system>
<testActions/>
</puffin> |
 |
Test actions
We now have the Puffin framework configured to the demo environment (or
your own if you have been altering these statements for your own Web
application). Now it's time to configure the possible test actions we
will use in our test plans. But first, let's review what a test action is.
A test action is the smallest executable element in a test plan. It may or
may not involve a call to your Web application, they usually do. It can
represent calls to the database or simply a step in a test plan in which
you process some data at a key point in the test plan's execution. Here, we will stick to
those test actions that involve a call to your
Web application.
Let's take a brief detour and discuss the demo application that comes
with Puffin. The Puffin Palace is a relatively simple Perl CGI application
in which you log in, select a type of item from a list of categories,
then select some number of items for your shopping cart, and check out and see the total cost. Though simple, we can use it to drive
Puffin through its paces.
We'll start by creating a test action that represents the application's
login. This is a simple, nonsecure login in name only. Basically, a user
would click a link, and the app would generate a session identifier and
incorporate it into all links on the first page (the list of product
categories). Here is the Puffin config file entry for the login
message.
Listing 8. Puffin config file entry for the login
message
<testActions>
<testAction name='login'>
<path>/puffindemoapp/start.cgi</path>
</testAction>
</testActions>
|
So this test action represents a user's clicking on some link that calls
the start.cgi page in the Puffin demo application: http://localhost/puffindemoapp/start.cgi.
When this page of the demo application is called, the following HTML is
returned.
Listing 9. HTML returned
<html>
<body>
<center>
MENU
<p/>
<a href="itemList.cgi?session_id=3201020311352&list=Pets"
>Pets</a><br/>
<a href="itemList.cgi?session_id=3201020311352&list=Books"
>Books</a><br/>
<a href="itemList.cgi?session_id=3201020311352&list=Gear"
>Gear</a><br/>
<a href="itemList.cgi?session_id=3201020311352&list=Posters"
>Posters</a><br/>
<p/>
<a href="showCart.cgi?session_id=3201020311352">View Cart</a>
<br/>
<a href="logout.cgi?session_id=3201020311352">Log out</a>
</center>
</body>
</html>
|
The demo application randomly generates the session_id parameter on the
query string for the itemList.cgi call. This session ID is then sent in all
subsequent calls to identify the user. In the background, the demo app is
creating a unique cart file, etc. The details are not relevant, other
than the fact that they are meant to mimic the actual login process for
some Web applications. We need to get that session ID, so Puffin can
send it if any other test actions are called after this one in a test
plan.
Our first output token
Puffin maintains this type of information in a TokenDictionary object, which
is basically a glorified hash of named values. Puffin can extract these
values from a given test action's response (or from other previously
existing tokens or from several other sources of information) and store
them in the token dictionary. The act of initializing this value is called
"processing an output" because most of the time, you will be retrieving this
value from the response document the Web application returns upon execution
of a test action; you will be retrieving an "output" of that test action's
execution.
How do we tell Puffin to extract this output (in our case, the session
ID) every time it executes this test action? Exactly as you'd guess: with
information in the Puffin config file.
Listing 10. (Partial) puffinConfig.xml, configuring the test action, continued
<testActions>
<testAction name='login'>
<path>/puffindemoapp/start.cgi</path>
<!-- OUTPUTS -->
<output name="SESSION_ID" processor="extractRegex">
<param name="expr" eval="0">
<![CDATA[href="itemList\.cgi\?session_id=(\d*)]]>
</param>
</output>
</testAction>
</testActions> |
The bold code above tells Puffin to use the extractRegex output
processor to extract a value from the response of the test action named
login and store it in the current token dictionary under the name
SESSION_ID. The processor takes a parameter named expr, which, in our case
looks like this: href="itemList\.cgi\?session_id=(\d*).
For those of you familiar with regular expressions (regexes), this is pretty
straightforward. For all others, this probably looks like an erroneous
entry. Though it is beyond the scope of this article to teach the
nuances of regexe, suffice it to say that they
allow you to parse text that by matching certain patterns (see Resources). In the above
case, the pattern can be roughly translated into the following pseudo
code: Return all the text that starts with 'href="itemList.cgi?session_id='
and is followed by some number of numeric characters.
The parentheses around the \d* denote the actual subsection of the
matching phrase we want to return. So in our example HTML, if given the
following:
<a href="itemList.cgi?session_id=3201020311352&list=Posters">Posters</a><br/>
|
The regular expression would match this:
href="itemList.cgi?session_id=3201020311352
|
And return this:
Puffin would then store this value in a token named SESSION_ID in the
current token dictionary, which is maintained throughout a test plan's
execution.
Those of you familiar with regexes will note that the above
regex matched several lines (not just one). You are right.
However, because we used the extractRegex and not the extractRegexList output processor, Puffin returned the first match. There are many other
things you can do with output processing, including generating random
numbers or strings, calculations using tokens, etc. All are in the
User Guide.
Once this test action gets executed in a test plan, we will have an
output token called SESSION_ID. Now what?
Our first input
Well, our next test action to be configured is one that when executed will
produce a list of available items for a given type. Following is the beginning
of its entry in the Puffin config file.
<testAction name='getPetsList'>
<path>/puffindemoapp/itemList.cgi</path>
</testAction>
|
As you can see, the getPetsList test action involves a call to the
itemList.cgi page. The itemList.cgi
page takes a parameter called list to indicate what type of
list we want the app to display (pets, books, gear, posters). In our case, we want a
list of available pets, so let's
set up an input for this test action.
Listing 11. Set up input for this test action
<testAction name='getPetsList'>
<path>/puffindemoapp/itemList.cgi</path>
<input name="list" type="GET" processor="VALUE">
<param name='value'>Pets</param>
</input>
</testAction>
|
The <input> element we added configures the getPetsList test
action to take an input. An input is some value that Puffin will send to
the Web application when it executes this test action. Puffin allows you to
use POST or GET to send a value, or
you can add a HEADER. In the above case,
we set the type attribute to GET.
This instructs Puffin to add this value
to the query string when it executes this test action.
The input's processor attribute tells Puffin how to get the value it
will send as an input. In the above case, this is simple. It will simply
use the value specified in a parameter as is (in this case the word
"Pets"). Now if Puffin executes this test action, it would be as if you typed the
following into your browser's address line: http://localhost/puffindemoapp/itemList.cgi?list=Pets.
Puffin adds the ?list=Pets to the address as a query string (because
this is a GET type input).
What if the user goes to this page and selects a pet for purchase? What
shopping cart will Puffin use? We need the session ID. But wait. We don't
know what that value will be when we execute this. No problem. We will
instruct Puffin to get the value from the token dictionary.
Listing 12. Get value from token dictionary
<testAction name='getPetsList'>
<path>/puffindemoapp/itemList.cgi</path>
<input name="list" type="GET" processor="VALUE">
<param name='value'>Pets</param>
</input>
<input name="session_id" type="GET" processor="DICT">
<param name='key'>SESSION_ID</param>
</input>
</testAction>
|
You can probably guess how this works. Basically, the DICT value for
the processor attribute instructs Puffin to retrieve a value from the
current token dictionary and to use that value for the input (again of type
GET). So if we'd run a test plan that executed the login test action before
this one (the login test action, recall, extracted the value for a
SESSION_ID output token) and the value retrieved was 3201020311352, executing the above test action would be like calling the following through
your browser.
http://localhost/puffindemoapp/itemList.cgi?list=Pets&session_id=3201020311352
|
Puffin looks up the current value of the SESSION_ID token and uses that
value to build the query string.
We're almost ready to start writing test plans. Let's go ahead and set
up a few more test actions. Make your <testActions> element look like
the following.
Listing 13. <testActions> element
<testActions>
<testAction name='login'>
<path>/puffindemoapp/start.cgi</path>
<output name="SESSION_ID" processor="extractRegex">
<param name="expr" eval="0">
<![CDATA[href="itemList\.cgi\?session_id=(\d*)]]>
</param>
</output>
</testAction>
<testAction name='getPetsList'>
<path>/puffindemoapp/itemList.cgi</path>
<input name="list" type="GET" processor="VALUE">
<param name='value'>Pets</param>
</input>
<input name="session_id" type="GET" processor="DICT">
<param name='key'>SESSION_ID</param>
</input>
</testAction>
<testAction name='getPostersList'>
<path>/puffindemoapp/itemList.cgi</path>
<input name="list" type="GET" processor="VALUE">
<param name='value'>Posters</param>
</input>
<input name="session_id" type="GET" processor="DICT">
<param name='key'>SESSION_ID</param>
</input>
</testAction>
<testAction name='tooLong'>
<path>/puffindemoapp/tooLong.cgi</path>
</testAction>
<testAction name='serverError'>
<path>/puffindemoapp/serverError.cgi</path>
</testAction>
</testActions>
|
The getPostersList test action is similar to the getPetsList test
action. The other two will help us demonstrate what happens when we run
into problems.
There are several things to notice:
- We are doing some
of this the hard way (setting up two different list test actions when we
could set up only one with a dynamically driven value for the
listparameter, for example). But I want to show how this works.
-
POST- and HEADER-type input work in a very similar fashion.
- There are many other input processors. Check them all out in the User Guide.
- Finally, we can also create response analyzers specific to individual test actions (you
will do this often in your test plans). We will tackle this in later
articles, as well.
Now for a test plan
So far, we have built a Puffin configuration file. Puffin will use this file
to know how to communicate with your Web application. However, it does
not know what communication you want it to execute. For this, you need a
test plan file. For now, rename the test.plan file that comes with Puffin
to test.planORIG or similar, delete the old test.plan file, and create a new
empty file of that name.
Puffin allows for two different test plan formats: a simple format that
allows you to make quick and dirty test plans that you don't expect to grow
in complexity and you may not even keep — and a complex, XML-based format
that allows for the following:
- Including plans in other plans
- Iterating over plans until some condition occurs
- Grouping multiple test actions into a task that gets executed as a
whole
- Dependencies between tasks (if one fails, then a dependent task is not
executed)
That's too much to tackle here. We will attack those in the next
article. We will focus here on simple test plans, so in your new test.plan
file, enter the following (with no preceding spaces), and save it in the
same folder as puffin.py.
# This is a sample test plan.
login
getPetsList
getPostersList
tooLong
serverError
|
As you might guess, Puffin will ignore any line starting with a #
character or a whitespace as not containing the name of a test action. It
will assume every other line contains the name of a test action and will
attempt to retrieve the test action's configuration information from the
current Puffin config file.
You can create as many test plans as you like with the test actions in
whatever order. There is no need to keep them called test.plan other than
convenience (other names or placement in other folders requires using the
--test.plan=... command-line argument when you run Puffin).
This is one of the fundamental strengths of Puffin. You configure a
given test action in one place. Then you can use it in as many test plans
as you like.
Running a test plan
You have a test plan. Now it's time to execute and see what this whole
smoke-test thing is all about. Here is all that is required:
- Make sure the demo application is running and that you can hit it using
your browser or other appropriate client of the Web application (remember
Puffin isn't just for Web sites — it can even test Web services).
- On the command line, navigate to the folder containing puffin.py and
type the following:>>
puffin.
If not specified, Puffin will use the following (enter puffin --help
for a full run-down on command-line arguments):
-
test.plan=test.plan
-
puffin.conf=puffinConfig.xml
-
logging=WARN
You should see the following:
Listing 14. Running a
test plan
Starting Puffin.
Initializing framework.
Beginning test plan execution.
1. Executing task: login.
----Test action: login --> SUCCESS.
Task result: SUCCESS.
2. Executing task: getPetsList.
----Test action: getPetsList --> SUCCESS.
Task result: SUCCESS.
3. Executing task: getPostersList.
----Test action: getPostersList --> SUCCESS.
Task result: SUCCESS.
4. Executing task: tooLong.
----Test action: tooLong --> FAILURE.
Task result: FAILURE.
5. Executing task: serverError.
----Test action: serverError --> FAILURE.
Task result: FAILURE.
Ending test plan execution.
Ending Puffin.
|
Note that at No. 4, Puffin will appear to hang. This is exactly right as
this script should take 30 seconds to execute (thus, failing in terms of the
TimerResponseAnalyzer). If you got an error, check the following:
- All Puffin requirements are installed properly on your machine
(Python, eGenix Base, PyXML, Puffin itself).
- Your puffinConfig.xml is both a valid XML document and
correct.
- You don't have any spurious spellings in your test.plan
file.
Here's what Puffin did:
- Loaded the config file (in this case the default) and initialized
itself.
- Loaded the test plan and initialized it with the information from
the configuration file.
- Executed each test action in the order it was listed in the test plan.
For each test action, this involves the following substeps:
- It processed any inputs.
- It made the call to the Web application and retrieved its
results.
- It analyzed the results using both the
StatusResponseAnalyzer and
the TimerResponseAnalyzer.
- It processed any outputs and placed their values into the current
token dictionary.
- Reported the status for each action on the command line (using the
StreamHandler, remember).
- Wrote a file (puffinResults.log) containing a full report of the entire
test plan's execution. If this file already exists, Puffin appends the
results onto the end of the file.
- For each failure, Puffin wrote out a failure alert to a file in this
case (puffinFailures.log). This could just as easily have been an e-mail
address or similar.
- Puffin closed down.
|
Conclusion
We've only scratched the surface, but here's a mental exercise that
may help prove the importance of what we covered. Suppose we now decide
that we must write the itemList page of our application using
JSPs instead of Perl CGI (certainly an all too common occurrence). All
that we would have to do to have our existing test plans continue to work
with Puffin is to change the configuration of the itemList test actions to
use itemList.jsp instead of itemList.cgi. Everything else — all the
existing test plans, etc. — would continue to work while you began your
conversion tasks.
Now imagine the same situation with 100 pages' worth of Web application
and tens of different test plans that all roll together into one, complex
end-to-end smoke test for the entire Web application. Now instead of
rewriting itemList.cgi, you are converting 30 or 40 CGI scripts, or worse,
several developers converting 30 or 40 CGI scripts. You can begin to
imagine why Puffin can be a lifesaver you never even knew you needed.
Play around with the test plan and Puffin configuration file we
created through this article and read the User Guide for more
information. You should be able to get up and running with your own Web
applications in short order.
Resources Learn
-
Puffin's logging system uses the
popular log4j design by Vinay Sajip.
-
For the programmer new to Python and regular expressions,"Charming Python:
Text processing in Python" summarizes Python's text processing facilities. The
article explains some general concepts of regular expressions and offers advice on
when to use (or not use) them while processing text.
-
More practical advice on using regular expressions to search for and modify patterns in
text is in the tutorial "Using regular
expressions." The tutorial starts with the basics, then progresses to intermediate
and advanced topics, with lots of examples along the way.
-
To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
-
Stay current with developerWorks' Technical events and webcasts.
-
Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
-
Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
Get products and technologies
-
Download and uncompress Puffin
at SourceForge.
-
Download and install the appropriate Python distribution for your platform.
-
Download and install the PyXML
package for your platform. This open source project includes support for many XML, XSL, and XPath functions not yet found in the Python distribution.
-
Download and install the eGenix base package
of Python extensions. eGenix.com provides many free and commercial add-ons for the Python
programming language. Their base package includes such
things as Calendar-like date-time support, an efficient Stack datatype and
several more built-in tools. Puffin uses eGenix for its excellent date-time
manipulation functionality.
-
Innovate your next open source development project with IBM trial software, available for download or on DVD.
-
Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
Discuss
About the author  | | | Keyton Weissinger is the founder and principal developer for Key Focus Inc. He writes and edits for O'Reilly & Associates. At his day job, he is the manager of the lodging development group for Radiant Systems in Atlanta, Georgia. |
Rate this page
| 
