2. Presentation Schedule
• Cliff - 10-20m on RUM
• Aaron – 10-20m on Monitoring in real-time
• Balaji – 10-20m on Correlating RUM and Business Analytics
• Final Q&A
4. shhhh…..
• We are not the
fastest retail site
on the internet
today.
IE 8 – Web Page Test Comp
Index – Item Page Performance
11s – Fully Loaded time
4
5. What gets measured, gets done…
• Before we start to optimize – see where our pain is
and prepare to measure for success.
• You never go on a diet without first stepping on the
scale…
5
Source: Flickr
6. Synthetic Monitoring
Pros
– Technology is great
– Real Browsers (IE, Chrome,
FF)
– Built in Alerting
– Charting/Trending
– Waterfall analysis
– Screen shots & headers
– Object level detail
Cons
– False Positives
– Impossible/impractical to
measure everything
– Fixed number of browser/OS
combinations
– Simulated bandwidth
constraints at best
– Too few data points for
statistical relevance
6
14. What we found scared us…
Home Page Performance – Jan 2012
14
15. Set some goals and SLAs
• Focus on ‘Page Processing Time’ First
– see Golden Rule – 80% of time spent here (more like 90% for Walmart)
• Look at backend & network response time SLAs later
• Use 95th
Percentile
• Set Achievable SLAs
• Revisit Monthly
• Celebrate Wins!
15
16. Case Study: Item Page
• Problem:
– Page takes ~24s for slowest 5% of users
• Too many elements
• Slow third party modules
• Several other pagespeed ‘no-nos’
• Goal: Meet SLA for February
– 20s (95th
percentile)
• Approach:
– Scrum team dedicated to perf optimization for 1 sprint
– Team pools resources and ideas - focuses on biggest bang
16
23. Where’s the data?
The initial incarnation of RUM @ WMT leveraged Akamai at the edge.
• data reliability issues
• data availability issues
• data “freshness” issues
It works… for certain values of work
Source: Flickr
Source: Flickr
23
27. Pretty Pictures
RAW DATA FORMAT
uswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0,
10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0,
11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0,
9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0,
9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0,
11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0,
10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0,
8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0,
14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0
uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0,
2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5,
2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5,
2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0,
2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0,
3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0,
3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5,
2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0
27
28. The Work Tomorrow…
The Good
•Metric Throughput
•Commodity Storage
•Commodity Infrastructure
The Bad
•More Metric Throughput
•Calculation Complexity
•Web Sockets (pretty)
•Metric Fan-out
28
30. Is Page Performance a Factor of
Site Conversion? And how big is
it?
February, 2012
v
s
31. 31
Walmart.com - Fun Facts
• Reach
–Millions of Shoppers/week.
–Billions of page requests/year - Spikes up to 1500%
–Billions of internal product search volumes/year
• Scale
–Millions of active product SKUs + Market Place
–Millions of pages indexed in search engines
• Complexity
–1/4th
of page contents served by partners, affiliates and Marketplace
–Multiple departments, 10+ checkout paths
Page Performance & Site Conversion – Feb 2012
32. 32
So, how do you monitor?...
Page Performance & Site Conversion – Feb 2012
33. 33
Few Industry Benchmarks…
• Factoid 1: Large eCommerce site extensively A/B
tested page performance and published a study
showing 100 millisecond delay = 1% drop in revenue
• Factoid 2: Search Engines A/B tested performance
and found that a 500 millisecond delay caused a 20%
drop in traffic.
• Factoid 3: In an experiment across multiple retailers,
a 1 second delay caused a 7% decline in conversion
Page Performance & Site Conversion – Feb 2012
34. 34
So, how big is it for Walmart.com?
Page Performance & Site Conversion – Feb 2012
35. 35
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
36. 36
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 – Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
37. 37
Impact of site performance on overall site conversion rate….
Baseline – 1 in 2 site visits had response time > 4 seconds
* Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds
* Overall average site load time is lower for the converted population (3.22 Seconds) than the non-
converted population (6.03 Seconds)
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
38. 38
@ Page level….
Page load time is lower for Buyers compared to Non-Buyers
* The Page load time is highest for certain pages - 6.38 secs when there was a conversion and
8.06 where there was no conversion.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
39. 39
@ Department level….
Department load time is lower for Buyers compared to Non-Buyers
* Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
40. 40
What about bounce?
Page Bounce Rate Vs Response Time
* Key pages have high bounce rates which correlates with high T_Page as well
* Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing
pages.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
41. 41
Bounce rates @ department level….
Department Bounce Rate Vs Response Time
* High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly
higher
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
42. 42
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
43. 43
Phase 2 – Targets for Page Performance……
Conversion & Bounce Rate Impacts Drives Prioritization
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
44. 44
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
45. 45
Phase 3 – Success Story….
First Win….and yes conversion had positive improvements…
Page Performance & Site Conversion – Feb 2012
46. 46
• Page speed matters for site conversion!
• Monitor real user performance in a “Big Data” way!!
• Every 1s improvement = Up to 2% increase in CVR
• 100ms improvement = Up to 1% incremental revenue
• SEO benefits for entry pages and reduce bounces
• Test & Learn - Target segments and run A/B Tests focused on
improving page performance
Key Highlights
Page Performance & Site Conversion – Feb 2012
47. We’re Hiring for Everything!!!!
47
Source: Flickr
http://www.walmartlabs.com/open-positions/
https://walmartstores.com/careers/apply/?ba=eCom
@cliffcrocker or @GoFastWeb