Automation has come a long way in the last 15 years. This presentation tells the story of my journey from no automation, to running in excess of 1000 tests a day. It shares my insights, failures, successes and lessons learned, so hopefully you can get there too.
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
Journey to 1000 Tests a Day
1. Journey to 1000 Tests a Day
Bruce McLeod
Tuesday, 15 November 11
2. DISCLAIMER
The views and content presented in this presentation
are my own and do not represent the views of any of my
employers, past or present.
All content is copyright 2011 Bruce McLeod
Creative commons attribution for images is given at the end
of this presentation
Tuesday, 15 November 11
6. Huge gains compared to
manual execution
• Win32 desktop application
• 5 different clients (95, 98 Beta, NT4, NT 351 and Citrix)
• 5 days work for 3 people could be done in 4 hours
• 97% efficiency improvement
Tuesday, 15 November 11
7. But there were problems
• Tests were not maintainable
• Spent days in maintenance and development
• They were brittle and fragile
• And when I left for my next job ...
Tuesday, 15 November 11
12. A 2001 test automation recipe
Ingredients
1 eager client
1 test application
14 test automators
Tuesday, 15 November 11
13. A 2001 test automation recipe
Method
Step 1 : Write an automation “framework”
Step 2 : Create a gui map
Step 3 : Write a test
Step 4 : Repeat step 4 another 200 times
Tuesday, 15 November 11
14. Lessons learnt
• A Successful project
• 200 tests automated in 6 months
• Team was a real dev. team with all the same issues
• Versioning
• Merging and branching
• Coding standards
• Individual differences in style and design
Tuesday, 15 November 11
19. Keyword driven testing
Object Action Data
Textfield (domain) Enter text <domain>
Textfield (username) Enter text <username>
Textfield (password) Enter text <password>
Button (login) Click One left click
l Object - Action - data syntax
l Automation libraries with excel front end
Tuesday, 15 November 11
20. Keyword driven testing
• Advantages
• Reduction in coding effort
• “Anyone” can write tests!!
• Challenges
• “Anyone” can write tests!!
• Each implementation is custom
• Maintenance effort moves from code into ...
Tuesday, 15 November 11
21. … Lots of Spreadsheets !!!
Tuesday, 15 November 11
27. Unexpected benefits
• We could now use 1st generation developer tools
• Visual Studio 2005
• Version control
• C# and .net instead of VBA
Tuesday, 15 November 11
28. Unexpected benefits
• Testers were now true peers to developers
• Same tools
• Same development language
• Same recruitment process
• Object oriented design
Tuesday, 15 November 11
30. A discovery & epiphany!
The “braidy tester” blog posts
Tuesday, 15 November 11
31. The “sparkle” stack
• Used to test Microsoft Expression
• Key principles
• Test cases written in customer view
• Verification and data separated from tests
• Application internals re-created in test code
Tuesday, 15 November 11
56. Given I am a logged in user
When I have a valid credit card
And I have 1 click purchasing enabled
Then I should see the message “your purchase was successful”
And I should be re-directed to the home page.
Tuesday, 15 November 11