Automated verification is becoming increasingly important. Getting a product from idea to customer as fast as possible in a Continuous Delivery, or a Deployment pipeline is crucial in more businesses than ever before. But how do we get a product through that pipe line, with high quality? Kristian will talk about how automated verification can get you there.

1. Why Automated Verification
Matters
Kristian Karl, Spotify
@kristiankarl

2. 1983 -Started programming
1994 -Testing and automation
1998 -Consultant
2004 -Discovered Model-Based
Testing
2010 -Test Manager at Spotify

4. Technology
Product
Design

6. Confidence
Why does automated verification
matter?

7. ● What can we check?
● What is important to check?
● How can we check it?
● How to involve everyone?
● How do we get ROI?

8. ● Dev environments
● Builds on pull requests and merge
● Static code analysis
● Code style checking
● Dynamic code analysis
● Verification
● Archiving artefacts
● Deployment
What can be automated?

9. Business
Logic Coverage
Execution
Time / Costs
Code
Coverage
Test
Type
UI
Integration
Unit Test
High
Medium
Low
Low
Medium
High
Long / High
Medium
Short /
Low

10. A human using her intelligence
and experience to critique and
explore a system under test.
Testing

12. Agile Testing

13. Check for
the
expected
Find
the
unexpected
Business
Technology
● Functional testing
● BDD
● Specification by
example
● Unit tests
● Component tests
● Exploratory testing
● Scenarios
● Usability testing
● Performance
● Security

14. Our approach at Spotify
Disclaimer: There are 100+
teams at Spotify, and they
all do things a bit different.
Some teams do it
very differently...

15. Code
change
Pre-merge
build and
tests
Code and
quality review
Risk
analysis
Exploratory
testing
Continuous
Delivery

16. Pre-merge
● Quick feedback loop
● The total time for the build needs to
be time boxed
● Style checking
● Static code analysis
● Unit tests
● Stable functional tests
● Stable integration tests
● Packaging

17. Post-merge
● All the steps from the pre-review
build
● Longer running functional and
integration tests
● System tests
● Non-functional tests, like:
- resource leaks
- performance

18. What has helped Spotify
doing automated
verification?

19. The Test Data Service

20. The Test
The Test
Data Service
Some Production
Service
Request:
● Premium user
● Not connected to Facebook
● Using feature flag X
Query for user(s) matching
The criterias
List a matching user(s)
Return selected user
Pick one user, and lock it,
Prevent any other test from
using that user.
Time

21. Model-based Testing
à la GraphWalker

22. App not
running
Authentication
form displayed
Home
page
Start app
Close
Exit
Valid credentials
Bad credentials
Logout

23. Our Test Interface(s)

25. Test automator role Developer role
Defines a testing API
Implements the testing API

26. Test Result Service

28. One test result index to rule them all - Many ways of visualizing

29. So what’s the lessons
learned so far?

30. Backend teams
● Backend teams work with smallish micro
services.
● High degree of automation on all parts of the
pipeline.
● No one but the team touches the team’s code.
● They have operational responsibility.
● Few defects found after definition of done.

31. Frontend [mobile & desktop] teams
● Building monoliths
● 200+ people involved
● Low degree of automated verification
● Low testability
● High number of defects are
found after definition of done.
That hurts productivity...

34. Tragedy of the
commons
Individuals acting independently and
rationally according to each other's
self-interest, behave contrary to the
best interests of the whole group

36. What can you do to improve?
● Ways of working are more important than tools
and frameworks.
● Zero bug policy, instead of huge backlog of bugs.
● Pairing and embed
● Everyone writes and maintain
automated verification.
● Testability will require
non-functionality.
● Failed verifications must be
fixed ASAP.

37. Automated verification matters, because it can
● Increase your productivity
● Give you confidence to deliver anytime
● Enable you work with
defect prevention, rather than defect
detection

38. References
● https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
● https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/
● https://dl.dropboxusercontent.com/u/1018963/Articles/SpotifyScaling.pdf
● https://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-
automation-pyramid
● http://testautomationpatterns.wikispaces.com/
● http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/
● https://www.youtube.com/watch?v=IGQmdoK_ZfY
● http://graphwalker.org/