1. Testautomation
10 (sometimes painful)
lessons learned
Michael Palotas
Head of Quality Engineering Europe
eBay International
March 2012

2. eBay Inc. Facts
Founded in 1995
Based in San Jose, California
27770 Employees worldwide

3. Who is eBay?

4. Platform Facts
104 million
active buyers and sellers worldwide
68 Billion USD
transaction volume 2011
2000 USD
transaction volume every second

5. Data Facts
2 billion page views
each day
75 billion database calls
each day

6. Mobile Facts
70,000,000
Downloads of the mobile app
5 billion USD
transacted 2011 on mobile devices
1 million items
are listed with using mobile apps every week

7. Who is behind this?
Dominik Dary
Staff Software Engineer in Test
email: ddary@ebay.com

8. Who am I?
Michael Palotas
Head of Quality Engineering Europe
email: mpalotas@ebay.com
http://ch.linkedin.com/in/michaelpalotas
39 years old
born and grew up in Germany
lives in Zürich, Switzerland
somewhat extroverted engineer
loves test automation
innovation enabler
I like to do the right things and things right

9. Lesson 1: Flip the testing triangle
Manual
Tests
Manual Tests
E2E
Tests
E2E Automation
Tests Integration Tests
Unit
Tests Unit Tests
Inspiration: http://patrickwilsonwelsh.com/?p=32

10. Lesson 2: Everyone needs to know what is
automated
Use Test Aspects** Advantages
• Test aspects briefly describe • Clear overview for all team
“why” and “what” has to be members about test coverage
tested, without describing • Simplicity helps you to do
“how” quick reviews and
• Enables for early testing walkthroughs with your product
because tests can be written at owner
the same time the user story is • Enables you to identify manual
created and automated test duplicates
• Test aspects are structurally • Traceability between test
derived cases and specification
**Swiss Testing Day Presentation 2010: “Mit Testaspekten früh, intelligent und redundanzfrei testen” Melanie Späth
Image Source: http://www.masternewmedia.org/images/social_software_impact_individual_organizations.jpg

11. Lesson 3: A Tool is not a Strategy
A test tool/testing framework won't
teach your testers how to test
A clear testing process is mandatory
for test automation
Test Aspects and modeling of the
Biz-Domain layer is the key success
factor for your test strategy

12. Lesson 4: Write the right tests
Source: http://farm1.static.flickr.com/119/269335955_03a0ff4b67_o.jpg

13. Lesson 4: Write the right tests
Automate the business critical use
cases of your customer
Focus on end-to-end testing rather
than UI component tests
Design testability into your
product

14. Lesson 5: Instant Feedback is essential
Development Test
D T D T D T D T D T D T D T D T

15. Lesson 6: Automation is software development
Skillset & Knowledge Engineering practice
• You need people with software • Task oriented development
development skills and an helps you to get things done
appetite for testing • Follow the basic rules like lean
• Automation mindset is essential development principles
• Good understanding of the • Design the test before coding it
architecture of the application • Do code reviews
under test is required • Your developers can give you
very good feedback about your
code quality

16. Lesson 7: Maintainability is king

17. Lesson 7: Maintainability is king
Technical Debt Lean Test Automation
• Activities that result in • Eliminate Waste Definition of Done
technical debt include • Build Quality In Reviews
• Postponing
documentation and
• Create Knowledge Document the essentials
writing tests • Defer Commitment Don’t build a framework
• attending to TODO • Deliver Fast
comments and
• Ignoring compiler and • Respect People Long Leash
static code analysis • Optimize the Whole
warnings
• Knowledge that isn't
shared around the
organization
• Code that is too confusing
to be modified easily

18. Lesson 8: Speak the same language as the
developers
Use the same programming
language as development
Developers can write or maintain
tests if needed
Enables for a cross functional team
where everyone can work on nearly
every task

19. Lesson 9: Invest into the test infrastructure
• Speed up and simplify
the creation of test
data
• Offer and use web
services
• Web based test data
creation tools can be
used in test
automation and
manual testing

20. Lesson 10: Manual testing is still very important
Automated tests only do what you have implemented:
at the same speed, same order, with exactly the same
mouse clicks
Manual testing allows you to go off the beaten path
Test automation in combination with exploratory
testing combine the best of both worlds
Quality = Manual + Automated Testing

21. Conclusion of our ten lessons learned
1. Flip the testing triangle 6. Automation is software
2. A tool is not a strategy development itself
3. Everyone knows 7. Maintainability is king
what’s automated 8. Speak the same
4. Write the right tests language as the
5. Instant Feedback is developers
essential 9. Invest into the test
Infrastructure
10. Manual testing is still
very important

22. Thank You!
Questions or Comments?