Writing and maintaining a suite acceptance tests that can give you a high level of confidence in the behaviour and configuration of your system is a complex task. In this talk Dave will describe approaches to acceptance testing that allow teams to: work quickly and effectively; build excellent functional coverage for complex enterprise-scale systems; manage and maintain those tests in the face of change, and of evolution in both the codebase and the understanding of the business problem. This answered the following questions, and more: How do you fail fast? How do you make your testing scalable? How do you isolate test cases from one-another? How do you maintain a working body of tests when you radically change the interface to your system?

1. Dave Farley
http://www.davefarley.net
@davefarley77
http://www.continuous-delivery.co.uk
Acceptance Testing for
Continuous Delivery

2. The Role of Acceptance Testing
Local Dev. Env.
Source
Repository

3. The Role of Acceptance Testing
Artifact
Repository
Local Dev. Env.
Deployment Pipeline
Commit
Production Env.
Deployment
App.
Commit
Acceptance
Manual
Perf1
Perf2
Staged
Production
Source
Repository
Acceptance
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.

4. The Role of Acceptance Testing
Artifact
Repository
Local Dev. Env.
Deployment Pipeline
Commit
Production Env.
Deployment
App.
Commit
Acceptance
Manual
Perf1
Perf2
Staged
Production
Source
Repository
Acceptance
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Component
Performance
System
Performance
Acceptance

5. The Role of Acceptance Testing
Artifact
Repository
Local Dev. Env.
Deployment Pipeline
Commit
Production Env.
Deployment
App.
Commit
Acceptance
Manual
Perf1
Perf2
Staged
Production
Source
Repository
Acceptance
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Staging Env.
Deployment
App.
Manual Test Env.
Deployment
App.
Component
Performance
System
Performance
Acceptance

6. What is Acceptance Testing?
Asserts that the code does what the users want.

7. What is Acceptance Testing?
An automated “definition of done”

8. What is Acceptance Testing?
Asserts that the code works in a “production-like” test environment.

9. What is Acceptance Testing?
A test of the deployment and configuration of a whole system.

10. What is Acceptance Testing?
Provides timely feedback on stories - closes a feedback loop.

11. What is Acceptance Testing?
Acceptance Testing, ATDD, BDD, Specification by
Example, Executable Specifications.

12. What is Acceptance Testing?
A Good Acceptance Test is:
An Executable Specification of the
Behaviour of the System

13. What is Acceptance Testing?
Unit Test CodeIdea
Executable
spec.
Build Release

14. What is Acceptance Testing?
Unit Test CodeIdea
Executable
spec.
Build Release

15. What is Acceptance Testing?
Unit Test CodeIdea
Executable
spec.
Build Release

16. So What’s So Hard?
• Tests break when the SUT changes (Particularly UI)
• Tests are complex to develop
• This is a problem of design, the tests are too tightly-coupled to the
SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.

17. So What’s So Hard?
• Tests break when the SUT changes (Particularly UI)
• Tests are complex to develop
• This is a problem of design, the tests are too tightly-coupled to the
SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.
Anti-Pattern!
Anti-Pattern!
Anti-Pattern!
Anti-Pattern!

18. Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to keep them working
• Separate Testing/QA team owning automated tests

19. Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to keep them working
• Separate Testing/QA team owning automated testsAnti-Pattern!

20. Who Owns the Tests?
Developers Own
Acceptance Tests!

21. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

22. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

23. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
API Traders
Clearing
Destination
Other
external
end-points
Market
Makers
UI
Traders

24. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

25. Public API FIX API
Trade
Reporting
Gateway
…FIX API
“What” not “How”
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

26. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
API Traders
Clearing
Destination
Other
external
end-points
Market
Makers
UI
Traders
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

27. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
FIX API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-APIFIX-API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case

28. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
FIX API
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-API

29. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
Test
Case
API
External
Stubs
FIX-APIUI FIX-API

30. Public API FIX API
Trade
Reporting
Gateway
…
“What” not “How”
API
External
Stubs
FIX-APIUI FIX-API
Test infrastructure common to all acceptance tests

31. “What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions, and so should start
and init the app.
• Acceptance Test deployment should be a rehearsal for
Production Release
• This separation of concerns provides an opportunity for
optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead

32. “What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions, and so should start
and init the app.
• Acceptance Test deployment should be a rehearsal for
Production Release
• This separation of concerns provides an opportunity for
optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead
Anti-Pattern!

33. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

34. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

35. Test Isolation
• Any form of testing is about evaluating something in controlled
circumstances
• Isolation works on multiple levels
• Isolating the System under test
• Isolating test cases from each other
• Isolating test cases from themselves (temporal isolation)
• Isolation is a vital part of your Test Strategy

36. Test Isolation - Isolating the System Under Test

37. Test Isolation - Isolating the System Under Test
External System
‘A’
External System
‘C’
System Under Test
‘B’

38. Test Isolation - Isolating the System Under Test
External System
‘A’
External System
‘C’
System Under Test
‘B’

39. Test Isolation - Isolating the System Under Test
External System
‘A’
External System
‘C’
System Under Test
‘B’

40. Test Isolation - Isolating the System Under Test
External System
‘A’
External System
‘C’
System Under Test
‘B’
?

41. Test Isolation - Isolating the System Under Test
External System
‘A’
External System
‘C’
System Under Test
‘B’Anti-Pattern!

42. Test Isolation - Isolating the System Under Test
System Under Test
‘B’
Test Cases
Verifiable
Output

43. Test Isolation - Validating The Interfaces

44. Test Isolation - Validating The Interfaces
External System
‘A’
External System
‘C’
System Under Test
‘B’

45. Test Isolation - Validating The Interfaces
External System
‘A’
External System
‘C’
System Under Test
‘B’

46. Test Isolation - Validating The Interfaces
External System
‘A’
External System
‘C’
Test Cases
Verifiable
Output
System Under Test
‘B’
Test Cases
Verifiable
Output
Test Cases
Verifiable
Output

47. Test Isolation - Validating The Interfaces
External System
‘A’
External System
‘C’
Test Cases
Verifiable
Output
System Under Test
‘B’
Test Cases
Verifiable
Output
Test Cases
Verifiable
Output

48. Test Isolation - Isolating Test Cases
• Assuming multi-user systems…
• Tests should be efficient - We want to run LOTS!
• What we really want is to deploy once, and run LOTS of tests
• So we must avoid ANY dependencies between tests…
• Use natural functional isolation e.g.
• If testing Amazon, create a new account and a new book/product for every test-case
• If testing eBay create a new account and a new auction for every test-case
• If testing GitHub, create a new account and a new repository for every test-case
• …

49. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation

50. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)

51. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)

52. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)

53. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery

54. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery

55. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234

56. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234
Continuous Delivery6789

57. • We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self):
self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234
Continuous Delivery6789
• Alias your functional isolation entities
• In your test case create account ‘Dave’ in reality, in the test infrastructure, ask the
application to create account ‘Dave2938472398472’ and alias it to ‘Dave’ in your test
infrastructure.

58. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

59. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

60. Repeatability - Test Doubles
External System

61. Repeatability - Test Doubles
External System
Local Interface
to External System

62. Repeatability - Test Doubles
External System
Local Interface
to External System
Communications
to External System

63. Repeatability - Test Doubles
External System
Local Interface
to External System
Communications
to External System
TestStub
Simulating External System
Local Interface
to External System

64. Repeatability - Test Doubles
External System
Local Interface
to External System
Communications
to External System
TestStub
Simulating External System
Local Interface
to External SystemProduction
Test
Environm
ent
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
lkjalskjlakjsdlkajsld j
lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
lkajsodiuqpwouoi la
]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
lkjalskjlakjsdlkajsld j
lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
lkajsodiuqpwouoi la
]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
Configuration

65. Test Doubles As Part of Test Infrastructure
TestStub
Simulating External System
Local Interface
to External System

66. Test Doubles As Part of Test Infrastructure
TestStub
Simulating External System
Local Interface
to External System

67. Test Doubles As Part of Test Infrastructure
TestStub
Simulating External System
Local Interface
to External System
Public Interface

68. Test Doubles As Part of Test Infrastructure
TestStub
Simulating External System
Local Interface
to External System
Public Interface

69. Test Doubles As Part of Test Infrastructure
TestStub
Simulating External System
Local Interface
to External System
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
Test Infrastructure
Back-Channel
Public Interface
System
UnderTest

70. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

71. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

72. Language of the Problem Domain - DSL
• A Simple ‘DSL’ Solves many of our problems
• Ease of TestCase creation
• Readability
• Ease of Maintenance
• Separation of “What” from “How”
• Test Isolation
• The Chance to abstract complex set-up and scenarios
• …

73. Language of the Problem Domain - DSL
@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
trading.selectDealTicket("instrument");
trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}

74. Language of the Problem Domain - DSL
@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
trading.selectDealTicket("instrument");
trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 50", "executionQuantity: 4");
}

75. Language of the Problem Domain - DSL
@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
trading.selectDealTicket("instrument");
trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 50", "executionQuantity: 4");
}
@Before
public void beforeEveryTest()
{
adminAPI.createInstrument("name: instrument");
registrationAPI.createUser("user");
registrationAPI.createUser("marketMaker", "accountType: MARKET_MAKER");
tradingUI.loginAsLive("user");
}

76. Language of the Problem Domain - DSL
public void placeOrder(final String... args)
{
final DslParams params =
new DslParams(args,
new OptionalParam("type").setDefault("Limit").setAllowedValues("limit", "market", "StopMarket"),
new OptionalParam("side").setDefault("Buy").setAllowedValues("buy", "sell"),
new OptionalParam("price"),
new OptionalParam("triggerPrice"),
new OptionalParam("quantity"),
new OptionalParam("stopProfitOffset"),
new OptionalParam("stopLossOffset"),
new OptionalParam("confirmFeedback").setDefault("true"));
getDealTicketPageDriver().placeOrder(params.value("type"),
params.value("side"),
params.value("price"),
params.value("triggerPrice"),
params.value("quantity"),
params.value("stopProfitOffset"),
params.value("stopLossOffset"));
if (params.valueAsBoolean("confirmFeedback"))
{
getDealTicketPageDriver().clickOrderFeedbackConfirmationButton();
}
LOGGER.debug("placeOrder(" + Arrays.deepToString(args) + ")");
}

77. Language of the Problem Domain - DSL
@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
tradingUI.showDealTicket("instrument");
tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
tradingUI.dealTicket.dismissFeedbackMessage();
tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 50", "executionQuantity: 4");
}

78. Language of the Problem Domain - DSL
@Test
public void shouldSupportPlacingValidBuyAndSellLimitOrders()
{
tradingUI.showDealTicket("instrument");
tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
tradingUI.dealTicket.dismissFeedbackMessage();
tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
}
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 50", "executionQuantity: 4");
}

79. Language of the Problem Domain - DSL
@Channel(fixApi, dealTicket, publicApi)
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 123.45", "executionQuantity: 4");
}

80. Language of the Problem Domain - DSL
@Channel(fixApi, dealTicket, publicApi)
@Test
public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
{
trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
"side: buy", "quantity: 4", "matched: 4", "remaining: 0",
"executionPrice: 123.45", "executionQuantity: 4");
}

81. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

82. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

83. Testing with Time
• Test Cases should be deterministic
• Time is a problem for determinism - There are two options:
• Ignore time
• Control time

84. Testing With Time - Ignore Time
Mechanism
Filter out time-based values in your test infrastructure so that they
are ignored
Pros:
• Simple!
Cons:
• Can miss errors
• Prevents any hope of testing complex time-based scenarios

85. Mechanism
Treat Time as an external dependency, like any external system -
and Fake it!
Pros:
• Very Flexible!
• Can simulate any time-based scenario, with time under the control of the test
case.
Cons:
• Slightly more complex infrastructure
Testing With Time - Controlling Time

86. Testing With Time - Controlling Time
@Test
public void shouldBeOverdueAfterOneMonth()
{
book = library.borrowBook(“Continuous Delivery”);
assertFalse(book.isOverdue());
time.travel(“+1 week”);
assertFalse(book.isOverdue());
time.travel(“+4 weeks”);
assertTrue(book.isOverdue());
}

87. Testing With Time - Controlling Time
@Test
public void shouldBeOverdueAfterOneMonth()
{
book = library.borrowBook(“Continuous Delivery”);
assertFalse(book.isOverdue());
time.travel(“+1 week”);
assertFalse(book.isOverdue());
time.travel(“+4 weeks”);
assertTrue(book.isOverdue());
}

88. Testing With Time - Controlling Time

89. Testing With Time - Controlling Time
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
public void someTimeDependentMethod()
{
time = System.getTime();
}
System
UnderTest

90. Testing With Time - Controlling Time
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
public void someTimeDependentMethod()
{
time = Clock.getTime();
}
System
UnderTest

91. Testing With Time - Controlling Time
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
public void someTimeDependentMethod()
{
time = Clock.getTime();
}
public class Clock {
public static clock = new SystemClock();
public static void setTime(long newTime) {
clock.setTime(newTime);
}
public static long getTime() {
return clock.getTime();
}
System
UnderTest

92. Testing With Time - Controlling Time
Test Infrastructure
Test
Case
Test
Case
Test
Case
Test
Case
System
UnderTest
include Clock;
public void someTimeDependentMethod()
{
time = Clock.getTime();
}
public void onInit() {
// Remote Call - back-channel
systemUnderTest.setClock(new TestClock());
}
public void time-travel(String time) {
long newTime = parseTime(time);
// Remote Call - back-channel
systemUnderTest.setTime(newTime);
}
Test Infrastructure
Back-Channel
public class Clock {
public static clock = new SystemClock();
public static void setTime(long newTime) {
clock.setTime(newTime);
}
public static long getTime() {
return clock.getTime();
}
System
UnderTest

93. Test Environment Types
• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:

94. Test Environment Types
• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:
@TimeTravel
@Test
public void shouldDoSomethingThatNeedsFakeTime()
…
@Destructive
@Test
public void shouldDoSomethingThatKillsPartOfTheSystem()
…
@FPGA(version=1.3)
@Test
public void shouldDoSomethingThatRequiresSpecificHardware()
…

95. Test Environment Types
• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:
@TimeTravel
@Test
public void shouldDoSomethingThatNeedsFakeTime()
…
@Destructive
@Test
public void shouldDoSomethingThatKillsPartOfTheSystem()
…
@FPGA(version=1.3)
@Test
public void shouldDoSomethingThatRequiresSpecificHardware()
…

96. Test Environment Types

97. Test Environment Types

98. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

99. Properties of Good Acceptance Tests
• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient

100. Production-like Test Environments

101. Production-like Test Environments

102. Production-like Test Environments

103. Production-like Test Environments

104. Production-like Test Environments

105. Production-like Test Environments

106. Production-like Test Environments

107. Make Test Cases Internally Synchronous

108. Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete

109. Make Test Cases Internally Synchronous
Example DSL level Implementation…
public String placeOrder(String params…)
{
orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params));
return waitForOrderConfirmedOrFailOnTimeOut(orderSent);
}
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete

110. Make Test Cases Internally Synchronous
Example DSL level Implementation…
public String placeOrder(String params…)
{
orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params));
return waitForOrderConfirmedOrFailOnTimeOut(orderSent);
}
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete

111. Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete
• If you really have to, implement a
“poll-and-timeout” mechanism in your test-infrastructure
• Never, Never, Never, put a “wait(xx)” and expect your tests to be
(a) Reliable or (b) Efficient!
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete

112. Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete
• If you really have to, implement a
“poll-and-timeout” mechanism in your test-infrastructure
• Never, Never, Never, put a “wait(xx)” and expect your tests to be
(a) Reliable or (b) Efficient!
• Look for a “Concluding Event” listen for that in your DSL to report
an async call as complete
Anti-Pattern!

113. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.

114. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance Acceptance Test
Environment

115. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance Acceptance Test
Environment
AA

116. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance Acceptance Test
EnvironmentA
A

117. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance Acceptance Test
Environment
Test Host
Test Host
Test Host
Test Host
Test Host
A
A

118. Scaling-Up
Artifact
Repository
Deployment Pipeline
Acceptance
Commit
Component
Performance
System
Performance
Staging Env.
Deployment
App.
Production Env.
Deployment
App.
Source
Repository
Manual Test Env.
Deployment
App.
Deployment Pipeline
Commit
Manual Test Env.
Deployment
App.
Artifact
Repository
Acceptance
Acceptance
Acceptance Test
Environment
Test Host
Test Host
Test Host
Test Host
Test Host
AA

119. Anti-Patterns in Acceptance Testing

120. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems

121. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing

122. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need

123. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
about them. Start with YOUR strategy and evaluate tools against that.

124. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
Acceptance Tests!!!

125. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
test environments.

126. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
test environments.
• Don’t include Systems outside of your control in your Acceptance Test Scope

127. Anti-Patterns in Acceptance Testing
• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data
that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
test environments.
• Don’t include Systems outside of your control in your Acceptance Test Scope
• Don’t Put ‘wait()’ instructions in your tests hoping it will solve intermittency

128. Tricks for Success

129. Tricks for Success
• Do Ensure That Developers Own the Tests

130. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”

131. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”

132. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”

133. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another

134. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable

135. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.

136. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems

137. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments

138. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case

139. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case
• Do Test for ANY change

140. Tricks for Success
• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case
• Do Test for ANY change
• Do Keep your Tests Efficient

141. Q&A
http://www.continuous-delivery.co.uk
Dave Farley
http://www.davefarley.net
@davefarley77