Selenium 2 and Thucydides for ATDD

From automated web tests to automated acceptance criteria

John Ferguson Smart is an advocate of open source tools that support and automate the software development lifecycle, especially in Agile environments. Now he presents Thucydides, his own testing library built on top of Selenium 2/WebDriver. As he demonstrates, Thucydides rethinks and extends the potential of ATDD; first by turning automated web tests into automated acceptance criteria, and then by documenting that criteria for collaborative, multidisciplinary teams.

Acceptance test driven development, or ATDD, is a collaborative practice where users, testers, and developers define automated acceptance criteria early in the development process. The team then uses that criteria to guide subsequent design and coding. The acceptance criteria document a common understanding of the problem domain and its proposed solution. In multidisciplinary teams, such documentation helps to ensure that everyone agrees about a project's requirements and implementation. When acceptance criteria are automated, they become part of the "living documentation" of the system, always accurate and up to date.

If you are writing a web application, it makes sense to have your automated acceptance tests exercise the application as a user would, via a web interface. Many web testing libraries do just that, including open source tools such as Selenium, Canoo WebTest, WebDriver, and the recent Selenium-WebDriver integration, Selenium 2. These tools focus on testing the details of web page interaction, and they're great at that.

Most automated acceptance tests do not completely fulfill their potential, however, in that they do not drive the development and documentation process as much as they could. As a proponent and practitioner of test-driven development and other lean, collaborative development methodologies, I wanted a testing framework that would let me express and report on acceptance tests at a high level, group tests by stories and features, and then drill down into the details as required. And so I created one.

In this article I introduce Thucydides, an open source ATDD framework built on top of Selenium 2.

Introducing Thucydides

Thucydides is a software testing library designed to help developers write automated acceptance tests and web-based automated acceptance criteria. Thucydides provides extensions and utilities that improve web testing and development on Selenium 2 in three ways:

  1. Organizing web tests into reusable steps, and mapping them back to their original requirements and user stories.
  2. Generating reports and documentation about acceptance tests. Each report contains a narrative description of a given test, including a short comment and screenshot for each step. Reports serve as living documentation, illustrating how an application implements the specified requirements.
  3. Providing high-level summaries and aggregations in each report. These give an overview of how many stories and requirements have been tested by automated acceptance tests, which in turn lets developers gauge the relative complexity of each story.

Different stakeholders need to be able to view acceptance test at different levels. For instance, a developer needs to know how a feature should interact from the perspective of the end user. This will start at a high level, expressing the general business goals behind a feature, but during the development process the developer will progressively drill down until she is specifying detailed page-level interactions.

A tester will have similar needs, and may also need to review test results to see how the application behaved under test. QA staff would generally rather not run the application through each scenario, but can benefit from seeing how features are implemented by reviewing narrative reports, illustrated with application screenshots.

A project manager is not so interested in the finer details. She will be more interested in the relative size and complexity of different features, and in the number of working tests for each feature -- a good indicator of progress. A business representative or project management office might want an even more high-level view: based on the number and complexity of the features defined and implemented so far, is the project still on track?

All of these aspects of ATDD extend beyond the page-centric nature of traditional automated web testing, and they require particular extensions to implement.

Thucydides is built on top of Selenium 2.0, the latest version of the very popular Selenium open source automated testing framework. Selenium 2.0 is the fusion of Selenium and WebDriver, and incorporates many improvements over Selenium 1, including a simpler, cleaner API, built-in support for the Page Objects design pattern, and excellent Ajax support.

Installation and setup

It's easiest to see the benefits of Thucydides in practice, so the remainder of this article will be a hands-on demonstration. If you want to follow along, you will need to clone this demo project from Github onto your local disk. You will also need to have Maven installed in your development environment.

The Thucydides binaries are available from the Sonatype OSS repositories, so you need to add them to your Maven settings.xml file. See the Thucydides project wiki for detailed information about setting up your environment and Maven project for Thucydides.

I've taken my example application and testing scenarios from the jobboard-webtests project, hosted on Github. The application under test, jobboard, was built using the Play framework v. 1.1.1. If you want to run the demo app, first install Play, then go to jobboard-demos-->jobboard, and type play run.

Once you have the jobboard application running, you can load up the jobboard-webtests project in your favorite IDE and start coding.

Defining requirements in Thucydides

Software projects typically aim to provide end-users with a number of high-level features (sometimes called functionalities or capabilities). A feature is a general value proposition relating to something the application can do for the end user, expressed in terms you might put on a product flyer or press release. For example, if you were developing an online recruitment website, you might include features such as "Look for jobs" or "Manage job ads."

Features are usually too big and unwieldy to implement in one step. In fact, software developers typically divide them into manageable chunks. In Agile projects, these chunks are expressed as user stories, which are short sentences that capture the essence of what the user wants from each feature. In other projects, more formal use cases serve the same purpose.

When automating acceptance tests with Thucydides, you can define the features and stories you intend to test in the form of a simple class structure. This lets you map tests back to the requirements that they verify in a lightweight and robust manner. In Listing 1, the @Feature annotation indicates a feature, and nested empty classes within a feature class indicate the stories or use cases that comprise that feature.

Listing 1. Defining a feature

public class JobBoardApplication {

        public class LookForJobs {
            public class JobSeekerBrowsesJobAds {}
            public class JobSeekerBrowsesJobsByCategory {}
            public class JobSeekerAppliesForJob {}
        public class ManageJobAds {
            public class EmployerPostsJobAd {}
            public class EmployerRenewsJobAd {}
            public class EmployerReviewsJobApplications {}

Although you don't have to model your requirements in this way to use Thucydides, it does produce a very useful record of both the progress of the project and the number of requirements currently being tested.

Automating acceptance criteria in Thucydides

It's a great start to define application features and stories, but we usually need more information before we can begin coding. In a traditional waterfall-style project, this would take the form of detailed specifications. For projects based on iterative, agile, or lean development methodologies, features are defined by discussing each story in detail, in order to better understand its requirements. In particular, this involves drawing up a list of acceptance criteria, or conditions that an application must satisfy for a requirement to be complete. Engaging the requirements process as a team -- with product owners, business analysts, developers, testers, and so forth all participating -- is essential. It helps to create a shared vision and common understanding of the requirements to be implemented.

Acceptance criteria can take many forms, but the best acceptance criteria tend to be quite high level, focused on what the user is trying to do rather than how he or she will do it. This makes the acceptance criteria more robust and easier to maintain; it also lets the team stay focused on the core value proposition of a feature, rather than its implementation details. (That part comes later.)

A narrative style often works well for acceptance criteria. Listing 2 shows an acceptance criteria written in the popular "given-when-then" structure.

Listing 2. A given-when-then styled acceptance criteria

    Scenario: A job seeker can apply for a job online
        Given the job seeker is looking for a suitable job
        When the job seeker finds a good job post
        And the job seeker applies for the job
        Then the job seeker should see a confirmation of the job application
        And the employer should be notified

Some teams strive to make acceptance criteria easier to understand by making them more concrete. The criteria in Listing 3 reads more like an example than a specification.

Listing 3. Specification by example

    Scenario: A job seeker can apply for a job online
        Given the job seeker is looking for a job as a Java developer
        When the job seeker find a job post for a Java Developer at Acme Consulting
        And the job seeker applies for the job
        Then the job seeker should see a confirmation of the job application
        And the employer (Acme Consulting) should be notified

In some cases, such as when calculations are involved, a table-based approach could be more concise. BDD (behavior-driven development) frameworks like Cucumber and Easyb provide good support for table-based acceptance criteria, as do parameterized tests in JUnit.

Web test automation

Not all acceptance criteria have to be automated using web tests. Some core business logic, such as financial calculations, can be effectively tested directly against application code. Testing via the user interace is an important part of the testing process for most applications, however, especially those built on rich UIs such as GWT or JQuery. Automated web tests also provides a form of living documentation, illustrating step-by-step how each feature is implemented from the viewpoint of the end user.

Writing automated tests in Thucydides

Automating acceptance criteria involves more than simply writing automated tests for each criteria. It also involves expressing the acceptance criteria in a form that can be easily understood by all team members. Although Thucydides supports conventional testing tools like JUnit, I prefer the expressive style of Cucumber and Easyb.

In Listing 4, I've used the Thucydides's Easyb integration to automate the acceptance criteria from Listing 2

Listing 4. Automating acceptance criteria with Easyb

using "thucydides"
    thucydides.tests_story JobSeekerAppliesForJob

    scenario "A job seeker can apply for a job online",
        given "the job seeker is looking for a job as a Java developer"
        when "the job seeker find a job post for a Java Developer at Acme Consulting"
        and "the job seeker applies for the job"
        then "the job seeker should see a confirmation of the job application"
        and "And the employer (Acme Consulting) should be notified"

This is a fairly ordinary story in Easyb, with the exception of two lines: the using 'thucydides' line at the top lets Easyb know that I want to use the Thucydides Easyb plugin; and thucydides.tests_story JobSeekerAppliesForJob maps the automated acceptance criteria to the story defined in Listing 2. This is the logic that enables Thucydides to group acceptance criteria by story and by feature, and thus provide aggregate reporting across all stories and features.

At the moment, the test in Listing 5 (below) doesn't actually do much -- it will be executed during upcoming automated test runs, but will be flagged as pending. Nonetheless, it's now part of the automated test suite. Having each test automatically executed with every build makes it much easier to keep tabs on what features have been implemented, and which are still pending. The test will also appear in subsequent Thucydides reports as a pending step, as shown in Figure 1.

Figure 1. A pending step

Thucydides Steps: Implementing automated acceptance criteria

Once you've automated an application's acceptance criteria the development process can begin. The acceptance criteria act as goal-posts for the software development team, providing a clear set of objectives for each feature.

1 2 3 Page 1
Page 1 of 3
How to choose a low-code development platform