Three open source Web service testing tools get high marks
Capable soapUI, TestMaker, and WebInject toolsets shine once you conquer their learning curves
Click for larger view.
TestMaker is a Web service testing application from PushToTest. It requires Java 1.4 (or later) to execute. Although I tested the other tools on Windows, I installed TestMaker 4.4 on Ubuntu Linux 6.10 to see what Web service testing on Linux was like. Installation was simple, and once I had specified a JAVA_HOME environment variable, TestMaker launched and ran with no problems.
TestMaker's tests are embodied in scripts called “test agents.” The product lives up to its name by providing an Agent Wizard that will read a WSDL definition and automatically create a skeletal test agent.
I should point out that TestMaker is not limited to testing Web services; it can also be used to test Web applications. Bundled with TestMaker is a network monitoring tool that can watch HTTP traffic between your browser and a target Web application, and generate test cases from the interaction. However, I did not experiment with this capability, since it has limited value in association with Web services, which are usually driven by a client application.
TestMaker test agents are written in Jython (Python written in Java). This forges a double-edged sword. On the one hand, TestMaker's scripts can be as powerful as your programming abilities allow. Jython can access all the Java libraries (and unleash all their attendant capabilities), as well as classes and methods provided with TestMaker. The largest of TestMaker's libraries is TOOL (Test Object Oriented Library), and it includes classes for handling all sorts of communication protocols: HTTP, HTTPS, SOAP, POP3, JDBC, and more. You can, therefore, create magnificently elaborate test cases that approach or surpass any client application the Web service is likely to be called by.
Click for larger view.
The skeletal test agent created by the Agent Wizard is extremely barren: It knows about the Web methods of the target service, and it will execute without error, but it doesn't actually perform any requests, responses, or tests of results. I found that I had to examine the source code of one of the example test agents to fill in the missing pieces.