Three open source Web service testing tools get high marks

Capable soapUI, TestMaker, and WebInject toolsets shine once you conquer their learning curves

Page 2 of 2

Projects are arranged hierarchically and contain one or more TestSuites, which contain one or more TestCases, which in turn contain one or more test steps. The actual work – sending requests, receiving responses, analyzing results, and altering test execution flow – happens at the test step level. TestCases gather and organize the steps need to perform a specific operation on the target. TestSuites gather TestCases into larger aggregates that exercise a particular area of a Web service (such as the operations necessary to order a book). You can create new TestSuites, TestCases, and test steps by right-clicking on the parent node in the project's tree and selecting New from the pop-up context menu.

Then soapUI determines a given test's success or failure by assertions that you attach to test responses. There are a half dozen assertions to choose from, ranging from a "simple contains" test -- which succeeds if a provided string is matched -- to "XPath matching," which succeeds if an arbitrarily complex XPath expression matches the response.

A test step is analogous to a line of code in a program. Currently, soapUI defines six test step types, the most common being a Request, which sends an HTTP request to a destination and accepts a response. Control flow through test steps can be modified by inserting a Conditional GoTo test step, essentially a collection of one or more Xpath expressions that examines the most recent response. The first expression in the collection that succeeds causes branching to an associated target test step.
Click for larger view.

The Groovy test step (I am not making that name up) is soapUI's most powerful. Here, Groovy refers to the lightweight Javalike scripting language. A Groovy test step executes whatever Groovy code you want to place in it, which means that it can do pretty much anything that can be done in Groovy. The Groovy code in such a test step has access to the soapUI framework. For example, a Groovy test step might read information from a database (via JDBC), compare that information with a previous test step's response, and alter the execution flow of the TestCase accordingly – perhaps even executing a different TestCase.

Aside from the functional testing, SoapUI will also unleash load tests on a Web service. Each load test consists of the execution of one or more TestCases, and can be tuned to simulate various scenarios. You can, for example, control whether the test executes for a specific amount of time or a specific number of iterations, whether the activity executes in “bursts” or varies linearly over time, and so on.

When the load test is complete, a Load Test Editor provides a raft of statistics for each TestCase: number of executions; minimum, maximum, and average execution time; and others. You can even examine the results graphically on the Statistics Diagram page.

It is easy to get soapUI running; it will take you almost no time to build a basic project and construct rudimentary tests. My one gripe with the tool is that there is no contextual help in the system, which makes figuring out what your options are in particular areas of the application difficult. Nevertheless, the documentation provided is quite good, so any initial confusion evaporates with continued use.

Click for larger view.

TestMaker is a Web service testing application from PushToTest. It requires Java 1.4 (or later) to execute. Although I tested the other tools on Windows, I installed TestMaker 4.4 on Ubuntu Linux 6.10 to see what Web service testing on Linux was like. Installation was simple, and once I had specified a JAVA_HOME environment variable, TestMaker launched and ran with no problems.

TestMaker's tests are embodied in scripts called “test agents.” The product lives up to its name by providing an Agent Wizard that will read a WSDL definition and automatically create a skeletal test agent.

I should point out that TestMaker is not limited to testing Web services; it can also be used to test Web applications. Bundled with TestMaker is a network monitoring tool that can watch HTTP traffic between your browser and a target Web application, and generate test cases from the interaction. However, I did not experiment with this capability, since it has limited value in association with Web services, which are usually driven by a client application.

TestMaker test agents are written in Jython (Python written in Java). This forges a double-edged sword. On the one hand, TestMaker's scripts can be as powerful as your programming abilities allow. Jython can access all the Java libraries (and unleash all their attendant capabilities), as well as classes and methods provided with TestMaker. The largest of TestMaker's libraries is TOOL (Test Object Oriented Library), and it includes classes for handling all sorts of communication protocols: HTTP, HTTPS, SOAP, POP3, JDBC, and more. You can, therefore, create magnificently elaborate test cases that approach or surpass any client application the Web service is likely to be called by.

On the other hand, you're going to need to know Jython to get the most out of TestMaker, or in other words, you'll need to know Python and Java. That is not necessarily a bad thing, but it does mean that the learning curve for TestMaker is steeper than the other tools.
Click for larger view.

The skeletal test agent created by the Agent Wizard is extremely barren: It knows about the Web methods of the target service, and it will execute without error, but it doesn't actually perform any requests, responses, or tests of results. I found that I had to examine the source code of one of the example test agents to fill in the missing pieces.

Once you've passed the learning hump, it's easy to create new tests by copying, pasting, and tweaking existing code. In addition, the user interface is a joy to work with. When you first start TestMaker, it opens a “QuickStart” window, from which you can launch the Agent Wizard, jump directly to a menu of the example test agents provided, or dip into the extensive documentation. Also, TestMaker's UI is a standard multiwindowed IDE, with navigation in the left pane, editing in the right pane, results in the lower right, and a class navigation view in the lower left.

TestMaker can be executed from the command-line, so that your test agents can be executed by an automation system. In addition, TestMaker bundles the Apache Axis TCPMonitor tool, which allows you to monitor HTTP exchanges on a specified port. This is useful when you need to examine the internals of request/response pairs to determine how to craft your Jython test code.

A commercial version of TestMaker adds XSTest, which provides performance and scalability testing, a monitoring dashboard that runs tests cases automatically and provides live results to a viewing console, reporting capabilities, and TestNetwork – which can execute test agents remotely, thereby allowing you to build “farms” of test agent servers that can exercise a target Web application en masse.

TestMaker's documentation is exceptional, and the tool has the look and feel of a professional application. However, it is difficult to master. Set aside plenty of time to go through the tutorials and examine the invaluable source code examples.


WebInject is a super-lightweight testing tool that can automate the testing of both Web services and Web applications. In fact, WebInject's ability to test XML/SOAP Web services appears to be a recent addition to the tool, as earlier versions could not readily handle the SOAP protocol.

Written in Perl, WebInject is primarily a command-line tool, though its author provides a thin Perl/Tk user interface that at least simplifies the execution of tests for those unwilling to spend too much time at the command prompt. If you're not familiar with Perl, don't panic. WebInject is built so that you can construct your tests without having to touch so much as a byte of Perl code.

WebInject is really an execution and reporting engine. Unlike the other tools, it has no IDE-style user interface, so tests must be written in an editor outside of the WebInject UI. This gives WebInject a less professional feel, but doesn't hamper the tool. I envision users of WebInject having directories filled with text files of various test “templates.” To add a new test case, the user just pops open his or her favorite editor, does some cutting, some pasting, and a bit of tweaking to alter the template to fit the specific circumstance, and ba-ding!, you've got a new test case.
Click for larger view.

When you launch WebInject, it reads an XML file containing a description of the test cases that the engine is to perform. Each test case is described by a set of attributes within an XML <case ... /> element. So, a simple test case that verifies that a specific Web method returns a list of book titles that includes My Antonia would look like this:



   description1 = “Verify My Antonia in list”





   addheader=”SOAPaction: urn:getBookList”

   verifypositive=”My Antonia”


The “id” attribute not only provides a unique name for the test case, it also identifies the execution order of test cases. The rest of the attributes are reasonably straightforward (once you've perused the documentation, at least). The value passed to the “postbody” attribute tells WebInject to go grab an XML file named “soapListTest.xml” in the current directory and use that to craft the SOAP content for the request.

If the result includes the string “My Antonia,” this test will succeed. WebInject provides three additional “verifypositive” attributes, and the value of each is treated as a regular expression. This means you can create quite complex verification specifications -- specifically, you can construct a test step that passes only after it has made it through the filters of four regular expressions, one for each “verifypositive” attribute. A test case element could also include four “verifynegative” attributes that work just the opposite of “verifypositive”: The testcase fails if any of the regular expressions match.
Click for larger view.

In essence, a WebInject “project” is nothing more than an XML file filled with a set of <case ... /> elements strung one after the other. WebInject's simple structure lets you build tests with amazing rapidity. You must, however, have a moderately good understanding of the mechanics of SOAP protocols as well as a tool that lets you generate and capture HTTP/SOAP requests and responses. You'll need the requests to build the POST body and the responses so that you can create proper “verifypositive” and “verifynegative” regular expressions to check for success or failure. I used the Web Service Toolkit add-on for Eclipse to grab requests and responses for WebInject; once I had gotten the hang of it, I fell easily into the groove of building test cases.

As each test case executes, WebInject's UI displays the status (pass or fail). You can configure WebInject to provide a complete dump of all HTTP requests and responses, an exceedingly useful feature that you will certainly call upon when you need to debug a failing test case.

In addition, the UI will optionally produce a real-time graph of the response (round-trip) time for each request/response pair, so you can use WebInject to build and monitor performance tests. And, WebInject provides a plug-in for MRTG (Multi Routing Traffic Grapher), a network monitoring and data collection tool, allowing you to execute and capture the results of test cases over a period of time as well as examine the data for patterns and trends.

WebInject's single greatest characteristic is its simplicity. Once you've gotten the hang of WebInject's XML commands, you can build, modify, and extend test cases quickly. The entire documentation consists of a one long Web page that can be read in a single sitting. However, that one page of documentation sometimes leaves you wondering how to proceed. Furthermore, you'll need a moderately good understanding of the SOAP protocol, as well as a tool to extract a Web service response's POST body, which you'll need to build test cases.

At your service

These three tools place themselves along the spectrum from quick and easy to complex and powerful. Do you need to assemble some code quickly for hurling tests at your Web service? WebInject is the logical choice; you'll be buffeting your Web service code in an afternoon. Do you need a high-end tool that lets you create powerful tests that can be extended to draw upon other system resources – the filesystem, databases, e-mail, and such? Then roll up your sleeves and plow into TestMaker. Grab a Jython manual first, though, and prepare yourself for some heavy lifting.

I prefer the middle balance struck by soapUI. The skeletal tests created by soapUI's wizard were easier to flesh out than those built by TestMaker. And, if I needed to do something elaborate and off the wall, I could always call upon soapUI's Groovy capabilities – funny name aside, they do their job well.

In terms of how these products compare to commercial Web service testing tools, I'd say it's a mixed bag. They are, of course, inexpensive (free), and work well for easy to moderately-difficult jobs; on the other hand, they're somewhat less user-friendly than commercial tools and if you need to do something complex, you have to build it yourself. TestMaker comes closest to looking like a commercial product, but having to learn Jython means that it takes longer to set up some tests than it would with, say, Mindreef's SoapScope. soapUI is a tad less professional looking, but makes up for it by allowing you to construct useable tests without having to program. WebInject, however, is definitely a developer's tool. You need to know SOAP to use it well, and it isn't going to be as capable as soapUI or TestMaker because its test cases are very much driven by templates.


InfoWorld Scorecard
Features (20.0%)
Value (10.0%)
Portability (15.0%)
Scalability (20.0%)
Documentation (20.0%)
Ease of use (15.0%)
Overall Score (100%)
Eviware SoapUI 1.6 9.0 9.0 9.0 9.0 9.0 9.0 9.0
PushToTest TestMaker 9.0 9.0 9.0 9.0 9.0 7.0 8.7
WebInject 8.0 9.0 9.0 8.0 8.0 8.0 8.3
| 1 2 Page 2