12. 12
Anatomy of Your Web App
Responsive web site Web Servers
• Desktop Users
• Mobile Users
• Same or similar content
o Adapting to different
screen sizes
Mobile optimized
• “m-dot” site for mobile
• Custom built for mobile devices
All have html, images, JavaScript, css, plus third-party content
13. 13
Native App
• Android – Java
• iOS – Objective C/Swift
• Mobile app framework
o Same source code
Hybrid App
• Mix of native code + html
All have backend calls for login/authentication, account
settings, database calls, transactions, plus third-party SDK
calls
Mobile Apps and Back End Services
WebSite/Mobile
Back end
14. 14
The Need for Load Testing
• Everyone runs load testing on their website, right?
o Load and Performance testing is often not done or too late
o Black Friday, Cyber Monday, Holiday failures
• What’s the need for mobile web load testing?
o Mobile is growing faster than desktop
o Responsive sites and mobile sites deliver different content to
mobile devices over mobile network
• What about mobile app load testing?
o Mobile app back ends serve different content to mobile apps
o Remember Pokemon Go? It wasn’t the app that failed.
15. 15
When is your Peak Season?
• Peak is year-round
o Super Bowl Sunday – 110 million viewers
o Bi-weekly or monthly payroll processing
o Valentine’s Day, President’s Day, Mother’s Day, Father’s Day
o Quarterly processing and filing
o U.S. Tax filing
o Spring, Summer, Fall & Winter season sales
o July 4th, Christmas in July
o Back to School
o Olympics
o Halloween
o Election Day
o Singles Day – world’s biggest shopping day, $9B in 12 hours
o Black Friday, Cyber Monday and Holidays
Back to top to repeat again next year!
21. 21
Shifting Left
• Agile means shorter development cycles
o Used to be months, now weeks
o Quality sign-off within sprint
• Developers and QA working together
o Developers write unit tests; some teams swap roles
• Performance testing is moving to left
o Meet performance SLA in all new code
• Automation with CI tools including Jenkins
o Code that meets quality goals gets promoted automatically
Testing is moving to the left, including unit, functional, and performance
22. 22
Shifting Right
• Performance testing in development
o Development integrating performance tests into SDLC
o Strong adoption of open source test tools, such as JMeter
• Performance engineers using commercial tools
o Execute performance tests at scale, from multiple geos
o Testing not just software but infrastructure
• Testing tools addressing both sets of users
o Adding JMeter support, Continuous Integration automation
Moving performance tests from development to production
30. 30
Test Continuously at Different Load
Test from Development to Integration to Staging to Production
31. 31
JMeter
• Hosted by Apache Software Foundation
o http://jmeter.apache.org
• Designed to load test functional behavior and measure
performance
o Web – HTTP, HTTPS; SOAP / REST; FTP; Database, more
• GUI design
• Plug-ins
• Tests as xml
Popular Open Source performance testing tool
37. 37
Results of Shifting Performance Testing Left
Using Continuous Integration tools to automate testing
• Run incremental performance tests automatically with
each iteration
• Establish a trend line for performance early
• Developers find code and architecture issues early
• Performance teams focus on end-to-end infrastructure
and user related testing
• Minimize production escapes and implement
production testing
38. 38
Questions, Resources & Next Steps
• Holiday Readiness
o soasta.com/holiday-readiness
• The Performance Beacon
o soasta.com/blog
o soasta.com/blog/2015-popular-posts-web-performance
• Webinars
o soasta.com/webinars
o Load Testing for Peak; RUM-Based Testing
• CloudTest Lite
o soasta.com/cloudtest: Free community version for 100 VUs
Resources
61. Walmart found…
Converted shoppers were served pages
that were 2X faster than pages served to
non-converted shoppers.
Non-buyers were served category pages that
were 2-3 seconds slower than category pages
served to buyers.
For every 100 milliseconds of improvement,
incremental revenue increased by up to 1%.
64. Conversion rate shrinks by about 50%
when load time for “browse” pages
increases from 1 to 6 seconds
65.
66. What is the Conversion Impact Score?
The Conversion Impact Score (CIS) is a relative score that ranks
page groups by their propensity to negatively impact conversions
due to high load times. For each page group, the Conversion
Impact Score is calculated using the proportion of overall requests
that are associated with that group, along with the Spearman
Ranked Correlation between its load times and number of
conversions. The Conversion Impact Score will always be a
number between -1 and 1, though scores much greater than zero
should be very rare. The more negative the score, the more
detrimental to conversions that high load times for that page
group are, relative to the other page groups.
67. How much impact does the performance of this
page have on conversions?
1. Product
2. Category
3. Shopping cart
4. Home
5. Search results
69. 69
Test in Production
Some issues will only be found at scale load on production systems
Don’t forget your third party providers: CDNs, external scripts
70. 70
Measure Mobile App Performance Under
Load
All mobile apps perform well when there is no load
71. 71
Now That the Test is Over
• Analyze results for future areas to address
o Weak or stress areas that aren’t likely this time but …
• Identify third parties that were affected by load
o Coordinate with suppliers to address issues
o Align schedules or at least be aware of their changes
Even if the test passed, you’re not done
72. 72
Issues Will Happen
• “I am 100% confident that everything we tested will work
just fine.” – Nordstrom Performance Leader
o When loads are different than what you modeled, may have
problems
o E.g. At Nordstrom, during a recent anniversary sale, mobile
sign-ins were 9x greater than anticipated
o Used RUM to get to the root cause of the issue
Realize that Issues Will Happen Where You Can’t Predict Them
74. 74
CloudTest – Load Testing for mobile web, apps
• Load testing from development to production
o Continuous testing at any scale, on-prem or in the cloud
• Real-time analytics with customizable dashboards
o Safely test in production with full visibility
• Run tests live or automatically
o Full hands-on control or hands-off automation
• Full end-to-end view
o Front-end performance to back-end systems
Fast, scalable, affordable load testing for performance optimization
75. 75
Questions, Resources & Next Steps
• Holiday Readiness
o soasta.com/holiday-readiness
• The Performance Beacon
o soasta.com/blog
o soasta.com/blog/2015-popular-posts-web-performance
• Webinars
o soasta.com/webinars
o Load Testing for Peak; RUM-Based Testing
• CloudTest Lite
o soasta.com/cloudtest: Free community version for 100 VUs
Resources