Acceptance tests are more A+S than T+G

When doing Acceptance Test Driven Development, one of the things that people find hard to understand at first is the nature of an acceptance test.

In the process of teaching it over the last few days, combined with my own experience and some of the discussions during the Agile Alliance Test Tools workshop I've found another way of helping people to understand how to write 'good' acceptance tests.

Essentially, the process involves specification by example - the idea being that you use examples of actual usage to derive requirements. These examples are expressed as automated tests. What many people don't seem to get at first is that these tests are not like 'traditional' automated tests. They aren't written at the same level.

Levels of Abstraction

Acceptance tests should be both readable by your customer and automated. An acceptance test that looks like this isn't what we're going for:


		browser.type("username", "antony");
		browser.type("password", "pass");
		browser.click("document.forms[0].elements['Log in']");
		
		browser.waitForPageToLoad(PAGE_TIMEOUT);
		
		assertTrue( browser.isTextPresent(
                           SUCCESSFUL_LOGIN_INDICATOR));

We want something more like:

-When the user signs in with username antony and password pass

-Then the user sees the welcome page

During the Agile Alliance Test Tools workshop, Kevin Lawrence provided this useful insight:

Goals:

I want to get some money from my bank account.

Activities

I identify myself to the bank.

I tell the bank how much money I want.

The bank gives me money.

Tasks

I put my card in the slot.

I tap in my pin and press enter.

I click the little buttons to say how much money I want.

I click 'no, I don't want a receipt'

I take my cash

I click 'I have no further transactions today'

I remove my card

Most automated testing tools deal with the task level, such as the vanilla version of Selenium (in case you were wondering).

With acceptance tests we want them to be written more at the 'activity' level - less at the task level, using the goal as the tests title. FIT (and other members of that family such as FitNesse) and the relatively recent Concordion allow you to remain at the activity level. In fact, David Peterson (who I met recently and I have to say is an extremely nice person to talk to) wrote Concordion because the developers and testers kept writing FIT tests at the task level and he wanted a tool that discouraged that and encouraged the writing of activity-level tests.

General vs. Specific

There is another dimension to acceptance tests. The underlying concept is to seek concrete examples from which we can derive the generalisations, we want acceptance tests to be more specific and less general.

For example, a general statement would be "valid username and invalid password" whereas a specific statement would be "username of 'antony' and a password of 'thisaintright'".

In reality, I find it useful to clarify intent in certain situations so I might want my example to be more specific but hint at the general. For example, I might say "a valid username of 'antony' and an invalid password of 'thisaintright'".

Combining the Two

So, what we want from our acceptance tests is for them to be more activity-level and specific. I've been asked "aren't they just like flows in a use case?". The answer is no - use cases are often written at the activity level yet tend to be quite general.

The dimensions I've discussed above also help to illustrate how 'traditionally' motivated automated testing tools (including the vanilla version of Selenium) tend to be more task-level and specific.

As for manual test scripts - well, I've seen many forms but in my experience people tend to write them more at the task level. Whether they are specific or general tends to vary.

Antony Marcano's illustration of desired levels and detail in Acceptance Test Driven Development Acceptance Tests (agile acceptance tests)

In Conclusion

When selecting a tool for an agile team to automate a user story's acceptance tests, think carefully about your choice. Make sure that your chosen tool-set supports writing tests that are more activity-level and specific. Remember, you aren't constrained to a single tool. With both FIT and Concordion you are able to abstract the task level detail using a specialised framework to drive your application. Selenium Remote Control (SeleniumRC), for example, is a programmatic API available in several languages (including Java and .NET) and can sit behind your human-readable tests in the fixtures - or better still be abstracted further, as I prefer to, behind what David Peterson calls a 'Scripting DSL'.

Comments

I should have mentioned that a literate API in whatever language you write your production code in is also feasible - so long as you support customers in reading it and make it transparent - i.e. easy for them to browse when they want to...

For example...

givenThat ( a(user).isRegisteredAs (new User("Antony","pass"));
when (the (user).performs( new LoginAs(user)));
then ( the (user).sees( view( WELCOME_PAGE)));

Enjoy!

Antony Marcano

Alan illustrates with an example of how he has been using a similar approach... great minds think alike methinks?

Antony Marcano

Antony,
You've raised some good points here, thank you. I struggled with the title for a while before guessing A+S means Activities + Specific and T+G means Tasks + General. Anyway from my perspective, acceptance tests should be activity based, and illustrate the general principle we're trying to test while using - of necessity - specific instances of test data (such as a specific combination of account name + password). I think you've captured my approach when you talk about capturing the intent so we probably agree :)

PS: In the diagram do you mean to use acceptant (http://www.thefreedictionary.com/acceptant) or acceptance? I think you mean the latter.

Julian Harty