3. A story about building an executable documentation system
• Large hedge fund COTS
implementation
• Crippling regression test burden
Born out of
necessity
• 6-8 month cycle daily release
• 80 times defect reductionResults:
• and again
• Same patterns automation
platform
Then did it
again…
Safia– Archetypes and Templates, a voyage in automating acceptance
3
4. The problem quantified
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
4
5. The confidence problem!
Where it came from
First Approach to Acceptance Testing:
UAT in a phase, after a period of development
5
8. Becoming acceptance driven…
Building Integrity in
8
Develop
Ready User
Story
Customer
Verification
Specify tests
/ examples
Stakeholder(s) & BA
Testers
Developers
Backlog
Demo /
Showcase
Automate
acceptance
tests
Session
Based
Testing
9. The initial effect of automated acceptance testing
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
9
10. Knowledge and experience to design a framework
• binding tests to investment apps
A set of design patterns
• from business specialists
Inject knowledge
• easy to customise
Vanilla acceptance tests
• tests as specifications
Living documentation
What we gained from this
10
11. Illogical Architecture Diagram
How Safia Works
House
Adaptor
System
Adaptor
Trading system
Maps SAFIA terminology to
system terminology
e.g.
Calypso, Beauchamps, Summi
t
SAFIA automates at the
server level via the API
Safia fixtures
API
Fitnesse fixtures
Maps investment house
terminology to SAFIA
terminology e.g.
Trade Capture = Trade Entry
11
12. e.g.
FX, Bond, Equities
IRS, FRA, CDS
e.g.
Run a P&L report
Update a legal entity
Amend SDI
Amend fees
Creating automated test cases using Safia
How Safia Works
Test Archetype
“Test Function”
Create a trade
Instrument
Template
IRS template
Test
Create a valid IRS trade
Apply Template &
Archetype
12
14. It brings benefits on different levels
Why use a framework like this
14
Benefits
Increased
collaboration
Accelerating
implementation
& learning
Anyone can
interact with it
Teaching &
coaching aid
Reduced Cost
of Ownership
16. The path to faster release
Real data from the field
40
67.5
100
78
23
15
12
5 3
0
28
64
151
217
311
371
393
479
545
601
0 0
25
106
284
371
393
479
545
601
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
February April August November February April May June July August
#TestSuites
ManDaysEffort
Month
Time taken for Release Regression testing
Man Days Effort for Regression Test # Test Suites # Automated Suites
16
17. Tests as specs, the principles of test design
How we wrote the tests
17
What makes a
good acceptance
test?
Self-documenting
About business
intentions
Concise and
granularA specification
not a script
Anyone can
understand it
Based on testable
statements
18. ~One third is system understanding
Reducing the Cost of Ownership
18
19. Points to consider when automating
19
Theory ‘aint practice
Start early & iterate frequently
Distil and simplify
Tests or Documentation?
DON’T PANIC !!!
21. Questions
Thank you for listening
21
Please evaluate our
presentation by using the
evaluation booklets
which you can find in
your conference bag.
Thank you!
22. SQS Group Limited
Mike Scott & Tom Roden
mike.scott@sqs.com
tom.roden@sqs.com
SQS Group Limited
7-11 Moorgate, London EC2R 6AF
Internet: www.sqs.com
Notas del editor
Borne out of necessity, not dreamed up in a lab
Story of building an automation solution – out of necessity in real world project, struggling with quality and agilityShow framework, but talk thru story of interesting points and learnings derived along the wayevolved thru rapid iteration as it was built, then gradually refined and re-used on similar systems, distilled and simplifying it to something useful, agnostic products and technologiesIt is centred on investment / trading platforms, but the principles and experiences are much more universal
Problem – R reached 100 man days– growing and would mean not running it allIntroduced automation, teaching ATDDThen agreed to do full at source automationRapid ROI – story goes on but first walkthrough the changes made
Problem faced by many agile teams – not quite solving acceptance testingFirst – separate business UAT team, defects slow all other work to a stand-stillTesting, particularly in this way is like shining a torch onto the product – narrow beam = focus depth in detail, wide beam = less depth on one thing but more wider coverageSo the business were sent into a room with their torch to find the product…..waiting…..
An amount of time, gore and screaming later, the product was in a roughly suitable state to go live (or rather the business were too battle weary by then)Not a production quality, shippable product until the very end (and then not really clear – wide beam dispersion on our testing torch)Long UAT phase, delayed release – worse still poor quality in live
The product come rolling down the hill into test (withdev’s and others running after it) – asked to push it back upAll you can do to stop the boulder from rolling down into the gutterTesting at the end always used to make me feel this way.
Main changes:Implementing Specification by Example c.f. GojkoSpecification workshopsAutomation at source (alongside dev) w/ ETDemo’s
Problem – R reached 100 man days – growing and would mean not running it allIntroduced automation, teaching ATDDThen agreed to do full at source automationRapid ROI – story goes on but first walkthrough the changes made
From better tests to templatesTemplatesArchetypesUsage examples
Agile acceptance testing = BDD automation (at source)Increased collab = building the right product (build integrity in), involving the whole team and BUSINESSTools cover a range of technologies, and are open sourceAdvantageous Versus traditional automation for numerous reasons (More robust, Quicker, Easier to maintain,Reliable source of system documentationGreat accelerator for automationUseful by-product ofSafia is as a teaching and coaching aid, showing people how to write fixtures and tests cleanly and to BDD principlesEasily accessible to anyone in an organisation, business or IT
The tests were completely robust right from the startNever duplicated anything, broke a load of tests nor had to refactor a bunch of stuffFirst design was perfect, never needed to revise itSomething rather than nothing – It can be daunting. Start with something, and then make it better. A simple incomplete test suite is far better than none at all.Iterate – First version will be imperfect and incomplete. Improve quality and completeness iterativelySooner rather than later – A simple test suite soon is more useful, to a more complex, more fully featured test suite 6 months or more from now.Lean start-up patterns – build-measure—refactor-learn-repeatChallengesTrusting the tests (when not reliable)Utilities (to manage dependencies and data, particularly for asynchronous systems)Example usage so not duplicatingWhat is isn’t:A silver bulletAbout toolsThe only way to do it
Was asked to get regression down to 0 people for 0 days i.e. full lights out automation.Was harder than imagined due to the challenges mentionedBuilding a continuous delivery pipeline means taking full control of dependencies including integration with other systemsIf you don’t design it into applications (as was the case here) it can be hard to make apps testable once they are built (this was a COTS implementation) – lesson work out what you need to measure in live and what you need to do to test an app as you build it, that will save time in the long run in testing, extension and maintenance
describe the behaviour being checked and test what it purports to Sometimes notbe about business rules and process, not implementation jargon, test namesbe focused on the specific thing being testedfail due to the assertion being testedNot cover too much at once to affect ease of reading but group similar tests P&L Test exampleBe a specification of business logic, not be a list of instructions login, go to screenof use to everyone who needs to use them– collective test ownershipClear, explicit expectation they are documentation
Metric to support our claim (rather than refute it – though we searched objectively).This illustrates the case (one of the best supporting evidence) for living documentation, writing tests as specs so all can use them – in the long term support of a product you need tests for various stakeholders that can eliminate a good chunk of this system understanding time and cost.
