- Steve Thair presented on whether the current model of load/performance testing is broken for modern web applications. - He discussed how Betfair separated load injection from performance measurement due to the complexity of their system. - The current model of load testing with waterfalls, single reports, and scripted user journeys is insufficient for continuous delivery and real user monitoring needs. - Thair advocated for cheaper and more continuous methods like session replay from logs and APM tools to align with modern development practices.

1. Is the current model of load
testing broken?
Steve Thair
Seriti Consulting
@TheOpsMgr

2. 2
Some background….
 My User Group – Dec 2011 -
http://www.meetup.com/London-Web-Performance-Group/
(c) Seriti Consulting, 2011

3. Delivery Alert Deviation
from
Team
normal?
Ready Fail Fail to
for Build meet
Test? SLA’s?
Deploy
Measurements
Environment
Continuous End User
Perf Trends
Load Response
Injection
For Each Data
Service/API Point
Usage response
Profile
Production
logs System
Perf Details
monitors
http://www.slideshare.net/sthair/continuous-integration-a-performance-engineers-journey
CONFIDENTIAL and not for reproduction without prior written consent. © of the Sporting Exchange Limited.

4. 4
The Killer Comment…
“We’ve had to look at separating load
injection from performance
measurement”.
- Andrew Harding, Betfair
http://www.seriticonsulting.com/blog/2011
/12/9/is-the-current-model-of-
loadperformance-testing-broken.html
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf

5. So why are you paying all 5
that money for that
What are key expensive brand-name
reasons behind load testing tool then?
that decision?
Surely there are
open source or
cheap cloud
tools if all you
want is load?
What other
issues might we
have with
What are you
“traditional” load
using to measure
testing models
Performance
in a Web 2.0
then?
world?
@TheOpsMgr #ukcmg #webperf

6. 6
Defining the “current model”
“Stephen didn’t explicitly define
what the current model of
load/performance testing is…”
 Alex Podelko
http://applicationperformanceengineeringhub.com/is-the-current-model-
of-loadperformance-testing-broken/
@TheOpsMgr #ukcmg #webperf

7. 7
“Current Model Straw Man”
(1) Waterfall Development Cycle
(2) Load Tool compiles the report
(3) Reporting at the end of the test
(4) Request / Response paradigm
(5) Well-defined customer journeys
(c) Seriti Consulting, 2011

8. 8
Testing in a CI/Agile world
 Performance testing initiated every time code is committed to
the CM repository (e.g. SVN, GIT etc)
 Betfair issue was that their system was complex with many
layers of caching etc so it took longer to “warm up” the
environment (to achieve a steady performance state) than
they had between check-ins…
 So they needed “continuous injection” to keep the
environment constantly warm…
 The test tool never “stopped” to “compile the report”…
 So any tool that “reported at the end” wasn’t as useful…
@TheOpsMgr #ukcmg #webperf

9. 9
Test tools in a APM & RUM world
 “traditional test tools” generally have the ability to deploy
agents to gather metrics from the target environment
 But the depth of analysis and correlation falls well below that
of modern Application Performance Management tools e.g.
AppDynamics
 APM tools offer a deeper insight and better event correlation
across tiers…
 And they are getting (much) cheaper…
@TheOpsMgr #ukcmg #webperf

10. APM Measurement
@TheOpsMgr #ukcmg #webperf

11. Google Real Time Analytics
@TheOpsMgr #ukcmg #webperf

12. 12
WebSockets
 HTTP 1.1
 Start Timer
 GET /index.html
 Response 200 OK
 Stop Timer
 WebSockets
 Start Timer
 Socket “upgrade”
 N many frames…
 Stop??? Bi-directional socket channel
@TheOpsMgr #ukcmg #webperf

13. 13
HTTP 2.0
 HTTP 1.1
 Sequential
 ordered
 HTTP 2.0 (& SPDY)
 Multiplexed over a
single connection
 Responses
returned out of
sequence…
 Hard to time!
http://stackoverflow.com/questions/10480122/difference-between-http-pipeling-and-http-multiplexing-with-spdy
@TheOpsMgr #ukcmg #webperf

14. 14
Website Complexity
 263,000 entry points…
@TheOpsMgr #ukcmg #webperf

15. 15
Visitor Flow
 How many paths thru a
website?
 Classic script-driven
approaches can’t hope to
address the complexity
 Network and log file
replay solutions?
@TheOpsMgr #ukcmg #webperf

16. 16
A PCAP Solution to Replay?
 POC solution based on Cloudmeter
Pion + custom scripts
 Read a PCAP (network capture)
 Identify the HTTP traffic
 Filter it (based on your requirements)
 Parameterise it (query strings, POST
parameters etc)
 Randomise inputs from SQL, CSV etc
 Replay it against a test environment
 i.e. change the base URL
 Amplify & rate throttle req/sec as
required
http://www.cloudmeter.com/pion/data-processing.php
@TheOpsMgr #ukcmg #webperf

17. 17
Personal Opinion!
YMMV…
So what’s the Answer?
 Use the cheapest method to generate load that you can
find…
 Move away from scripting-based approaches towards using
real-user session replay (if possible!)
 Generate load continuously
 Measure continuously
 using APM & RUM type tools – FOSS or Commercial
 Look for changes in histograms, averages, standard dev etc
 Protocol and Framework aware instrumentation
 AFAIK this currently doesn’t exist…
@TheOpsMgr #ukcmg #webperf

18. 18
@LDNWebPerf User Group!
 Join our London Web Performance Meetup
 http://www.meetup.com/London-Web-Performance-Group/
 Next Wednesday 17th Oct – 7pm – Central London
 Follow us on Twitter @LDNWebPerf
 #LDNWebPerf & #WebPerf
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf

19. 19
About Me
 21yrs IT experience.
 Started with www in 1998 (IIS3! Site Server 3!).
 Web Architect @ BNP Paribas, CSFB etc
 Web Operations Manager for www.totaljobs.com, www.tes.co.uk
 Professional Services Manager @ www.siteconfidence.com
 Seriti Consulting – specialising in web operations, management and
Performance
 e:stephen.thair@seriticonsulting.com
 m:+44 7971 815 940
 Twitter: http://twitter.com/TheOpsMgr
 Blog: http://www.seriticonsulting.com/blog/
 LinkedIn: http://uk.linkedin.com/in/stephenthair
 Skype: seriti-steve
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf

20. 20
Questions?
(c) Seriti Consulting, 2011 @TheOpsMgr #ukcmg #webperf