Andrew Harding from Betfair's presentation on web performance testing in a continuous integration environment. Covers some good reasons why and why not to do perf testing during continuous integration.

1. London Web Meetup – Dec 2011
Continuous Integration – A Performance Engineers Tale
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

2. “WE WILL DO
CONTINUOUS
INTEGRATION commandment - The
1128 TESTING”
Management
th
2 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

3. INITIAL THOUGHTS
•Automate all of our Performance Tests
• Add tests to the Build process
CI 
3 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

4. WHY IT WONT WORK
•Performance testing needs a functional
application
• The length of time needed to gather
significant amount of data points
• Performance testing requires analysis of
results to find meaningful conclusions
4 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

5. MORE INFORMATION
5 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

6. GOALS FOR PERFORMANCE
•Measure Performance as part of every build
• Compare between builds (regression)
• Alert if performance deviates from normal
•Fail the build if performance does not meet
SLA’s
• Automatic process
6 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

7. Delivery
Team
Build
7 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

8. Delivery
Team
Ready
for Build
Test?
Deploy
Environment
8 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

9. Delivery
Team
Ready
for Build
Test?
Deploy
Environment
Performance
Perf Trends
Measurements
9 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

10. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Environment
Performance
Perf Trends
Measurements
10 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

11. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Environment
Continuous Performance
Perf Trends
Load Measurements
Injection
Usage
Profile
Production
logs
11 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

12. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Environment
Continuous End User
Perf Trends
Load Response
Injection
Service/API
Usage response
Profile
Production
logs System
monitors
12 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

13. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Environment
Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
13 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

14. “THE POINT OF TESTING IS TO FLUSH OUT, UNDER
CONTROLLED CONDITIONS, ANY PROBLEM THAT THE
SYSTEM WILL HAVE IN PRODUCTION. A SIGNIFICANT
PART OF THIS IS THE ENVIRONMENT WITHIN WHICH
THE PRODUCTION SYSTEM WILL RUN. IF YOU TEST IN
A DIFFERENT ENVIRONMENT, EVERY DIFFERENCE
RESULTS IN A RISK THAT WHAT HAPPENS UNDER TEST
WON’T HAPPEN IN PRODUCTION”
MARTIN FOWLER
14 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

15. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Production Measurements
Environment
End User
Perf Trends
Response
For Each Data
Service/API Point
response
System
Perf Details
monitors
15 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

16. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Environment
Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
16 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

17. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Environment
Continuous End User
Perf Trends
Load Response
Injection LoadRunner
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
17 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

18. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements Keynote
Environment
Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
18 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

19. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Dynatrace Ajax Premium
Environment
Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
19 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

20. ADVANTAGES
•Allows standard performance regression
testing to be fully automated, freeing up time
for more complex performance work
•Separating load injection from performance
measurements enables performance to be
included in build cycle
•Provides immediate feedback to delivery
teams
20 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

21. NOT IN SCOPE
•Capacity Testing
• Soak Testing
• Failure Testing
•Stress Testing
Still Plenty of Traditional work
to do
21 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

22. THINGS I LEARNT
•Keep an open mind to new stuff
22 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

23. THINGS I LEARNT
•Keep an open mind to new stuff
• Give it more than 2 minutes of thought
23 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

24. THINGS I LEARNT
•Keep an open mind to new stuff
• Give it more than 2 minutes of thought
• Get past the reasons it won’t work
24 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

25. THINGS I LEARNT
•Keep an open mind to new stuff
• Give it more than 2 minutes of thought
• Get past the reasons it won’t work
•Use Google
25 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

26. THINGS I LEARNT
•Keep an open mind to new stuff
• Give it more than 2 minutes of thought
• Get past the reasons it won’t work
•Use Google
• Design for the ideal, compromise later
26 CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

27. THANK YOU
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.