TestMaker test agents are written in Jython (Python written in Java). This forges a double-edged sword. On the one hand, TestMaker's scripts can be as powerful as your programming abilities allow. Jython can access all the Java libraries (and unleash all their attendant capabilities), as well as classes and methods provided with TestMaker. The largest of TestMaker's libraries is TOOL (Test Object Oriented Library), and it includes classes for handling all sorts of communication protocols: HTTP, HTTPS, SOAP, POP3, JDBC, and more. You can, therefore, create magnificently elaborate test cases that approach or surpass any client application the Web service is likely to be called by.

Click for larger view.

On the other hand, you're going to need to know Jython to get the most out of TestMaker, or in other words, you'll need to know Python and Java. That is not necessarily a bad thing, but it does mean that the learning curve for TestMaker is steeper than the other tools.

The skeletal test agent created by the Agent Wizard is extremely barren: It knows about the Web methods of the target service, and it will execute without error, but it doesn't actually perform any requests, responses, or tests of results. I found that I had to examine the source code of one of the example test agents to fill in the missing pieces.

Once you've passed the learning hump, it's easy to create new tests by copying, pasting, and tweaking existing code. In addition, the user interface is a joy to work with. When you first start TestMaker, it opens a “QuickStart” window, from which you can launch the Agent Wizard, jump directly to a menu of the example test agents provided, or dip into the extensive documentation. Also, TestMaker's UI is a standard multiwindowed IDE, with navigation in the left pane, editing in the right pane, results in the lower right, and a class navigation view in the lower left.

TestMaker can be executed from the command-line, so that your test agents can be executed by an automation system. In addition, TestMaker bundles the Apache Axis TCPMonitor tool, which allows you to monitor HTTP exchanges on a specified port. This is useful when you need to examine the internals of request/response pairs to determine how to craft your Jython test code.

A commercial version of TestMaker adds XSTest, which provides performance and scalability testing, a monitoring dashboard that runs tests cases automatically and provides live results to a viewing console, reporting capabilities, and TestNetwork – which can execute test agents remotely, thereby allowing you to build “farms” of test agent servers that can exercise a target Web application en masse.

TestMaker's documentation is exceptional, and the tool has the look and feel of a professional application. However, it is difficult to master. Set aside plenty of time to go through the tutorials and examine the invaluable source code examples.

WebInject

WebInject is a super-lightweight testing tool that can automate the testing of both Web services and Web applications. In fact, WebInject's ability to test XML/SOAP Web services appears to be a recent addition to the tool, as earlier versions could not readily handle the SOAP protocol.