Walmart proves the obvious, devknob wonders why people don't understand why page speed matters. This has been true and known to be true since the beginning of the internet. Do you think people won't get distracted easily and bounce when they're surfing on 2g, 3g and even 4g connections? Page speed matters, devknob is probably the best page speed optimizer in the world so if you need conversion optimization, you may want to visit devknob online at devknob.com
2. Presentation Schedule
• Cliff - 10-20m on RUM
• Aaron – 10-20m on Monitoring in real-time
• Balaji – 10-20m on Correlating RUM and Business Analytics
• Final Q&A
4. shhhh…..
• We are not the
fastest retail site
on the internet
today.
IE 8 – Web Page Test Comp
Index – Item Page Performance
11s – Fully Loaded time
4
5. What gets measured, gets done…
• Before we start to optimize – see where our pain is
and prepare to measure for success.
• You never go on a diet without first stepping on the
scale…
5
Source: Flickr
6. Synthetic Monitoring
Pros
– Technology is great
– Real Browsers (IE, Chrome,
FF)
– Built in Alerting
– Charting/Trending
– Waterfall analysis
– Screen shots & headers
– Object level detail
Cons
– False Positives
– Impossible/impractical to
measure everything
– Fixed number of browser/OS
combinations
– Simulated bandwidth
constraints at best
– Too few data points for
statistical relevance
6
14. What we found scared us…
Home Page Performance – Jan 2012
14
15. Set some goals and SLAs
• Focus on ‘Page Processing Time’ First
– see Golden Rule – 80% of time spent here (more like 90% for Walmart)
• Look at backend & network response time SLAs later
• Use 95th
Percentile
• Set Achievable SLAs
• Revisit Monthly
• Celebrate Wins!
15
16. Case Study: Item Page
• Problem:
– Page takes ~24s for slowest 5% of users
• Too many elements
• Slow third party modules
• Several other pagespeed ‘no-nos’
• Goal: Meet SLA for February
– 20s (95th
percentile)
• Approach:
– Scrum team dedicated to perf optimization for 1 sprint
– Team pools resources and ideas - focuses on biggest bang
16
23. Where’s the data?
The initial incarnation of RUM @ WMT leveraged Akamai at the edge.
• data reliability issues
• data availability issues
• data “freshness” issues
It works… for certain values of work
Source: Flickr
Source: Flickr
23
27. Pretty Pictures
RAW DATA FORMAT
uswmt.all.t_page.upper_95(9289.0),1329259510,1329260710,10|15904.0,9184.0,9125.0,12736.0,11735.0,16776.0,8484.0,
10839.0,14620.0,7579.0,8871.0,8240.0,12390.0,5211.0,10301.0,24784.0,9410.0,16554.0,9609.0,11871.0,12751.0,9797.0,
11003.0,15962.0,7953.0,7707.0,4181.0,11616.0,11746.0,12814.0,10566.0,24782.0,18303.0,20904.0,7718.0,8531.0,7312.0,
9614.0,8749.0,11671.0,5989.0,9832.0,10592.0,11611.0,16946.0,18858.0,14360.0,15927.0,10470.0,10140.0,11307.0,9739.0,
9772.0,9875.0,13641.0,11626.0,14758.0,6529.0,11727.0,10194.0,8003.0,10639.0,7297.0,9891.0,10312.0,12497.0,11557.0,
11406.0,12456.0,12939.0,11029.0,10813.0,11737.0,10618.0,14128.0,16879.0,15865.0,6255.0,14605.0,8861.0,27425.0,
10948.0,19666.0,7185.0,13266.0,13156.0,15111.0,13110.0,15151.0,8666.0,16775.0,10110.0,10387.0,17274.0,22183.0,
8937.0,13168.0,12267.0,11891.0,9635.0,10446.0,8129.0,9550.0,9229.0,8375.0,8657.0,11119.0,6799.0,9094.0,21952.0,
14989.0,16828.0,9001.0,13444.0,10332.0,13609.0,9266.0,13349.0,11546.0,9289.0
uswmt.all.t_page.median(1844.0),1329259510,1329260710,10|4165.0,2333.5,2073.5,2584.0,2547.0,2627.5,2401.0,1575.0,
2170.0,1169.0,1970.0,1838.0,2083.0,5211.0,2496.0,3242.5,1541.0,1437.5,1928.0,1971.0,1776.0,3108.0,2010.5,2044.0,2325.5,
2640.0,1733.0,3924.0,2629.0,1867.5,1782.0,2370.5,2921.0,4783.0,2260.0,1340.0,3256.0,2297.0,2565.0,1874.0,2000.0,2483.5,
2705.5,2432.0,1809.0,2826.0,2204.0,2695.0,1045.0,1615.5,2250.0,2387.0,1562.5,1998.0,2512.0,2139.0,1482.0,2138.5,2100.0,
2583.0,2652.0,3277.0,2549.0,1755.0,2196.5,2766.0,2989.5,3638.0,3034.0,3615.5,2650.5,5207.5,3023.0,1941.0,1918.5,1768.0,
3048.0,1522.5,2710.0,1392.0,2402.0,2005.0,3246.0,1383.0,1880.0,2398.0,1833.0,2579.0,2052.0,2622.0,2089.0,1102.0,1296.0,
3339.0,2132.5,2831.0,3466.0,2131.0,2026.0,2754.0,3228.5,1000.0,2075.0,2011.5,2428.0,4019.5,2788.0,1665.0,1968.0,2695.5,
2873.0,1752.0,2314.5,1766.0,2971.0,3091.5,2205.5,3033.0,2476.0,1844.0
27
28. The Work Tomorrow…
The Good
•Metric Throughput
•Commodity Storage
•Commodity Infrastructure
The Bad
•More Metric Throughput
•Calculation Complexity
•Web Sockets (pretty)
•Metric Fan-out
28
30. Is Page Performance a Factor of
Site Conversion? And how big is
it?
February, 2012
v
s
31. 31
Walmart.com - Fun Facts
• Reach
–Millions of Shoppers/week.
–Billions of page requests/year - Spikes up to 1500%
–Billions of internal product search volumes/year
• Scale
–Millions of active product SKUs + Market Place
–Millions of pages indexed in search engines
• Complexity
–1/4th
of page contents served by partners, affiliates and Marketplace
–Multiple departments, 10+ checkout paths
Page Performance & Site Conversion – Feb 2012
32. 32
So, how do you monitor?...
Page Performance & Site Conversion – Feb 2012
33. 33
Few Industry Benchmarks…
• Factoid 1: Large eCommerce site extensively A/B
tested page performance and published a study
showing 100 millisecond delay = 1% drop in revenue
• Factoid 2: Search Engines A/B tested performance
and found that a 500 millisecond delay caused a 20%
drop in traffic.
• Factoid 3: In an experiment across multiple retailers,
a 1 second delay caused a 7% decline in conversion
Page Performance & Site Conversion – Feb 2012
34. 34
So, how big is it for Walmart.com?
Page Performance & Site Conversion – Feb 2012
35. 35
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
36. 36
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 – Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
37. 37
Impact of site performance on overall site conversion rate….
Baseline – 1 in 2 site visits had response time > 4 seconds
* Sharp decline in conversion rate as average site load time increases from 1 to 4 seconds
* Overall average site load time is lower for the converted population (3.22 Seconds) than the non-
converted population (6.03 Seconds)
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
38. 38
@ Page level….
Page load time is lower for Buyers compared to Non-Buyers
* The Page load time is highest for certain pages - 6.38 secs when there was a conversion and
8.06 where there was no conversion.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
39. 39
@ Department level….
Department load time is lower for Buyers compared to Non-Buyers
* Key Categories has 2-3 seconds difference b/w buyer Vs non-buyer
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
40. 40
What about bounce?
Page Bounce Rate Vs Response Time
* Key pages have high bounce rates which correlates with high T_Page as well
* Significant difference ( up to 9secs) in T_Page between bounced and non-bounced for landing
pages.
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
41. 41
Bounce rates @ department level….
Department Bounce Rate Vs Response Time
* High T_Page for key pages (up to 19.82s) and key department making Bounce rate significantly
higher
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
42. 42
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
43. 43
Phase 2 – Targets for Page Performance……
Conversion & Bounce Rate Impacts Drives Prioritization
Note: Load Time here is the time taken from head of the page to page ready (T_Page)
Page Performance & Site Conversion – Feb 2012
44. 44
Agenda
• Phase 1 – Baseline Measurement - Impact of Site
Performance on Conversion, Bounce rates &
Revenue
• Phase 2 - Targets for Page Performance
• Phase 3 – Optimization Results
• Key Highlights & Takeaways
Page Performance & Site Conversion – Feb 2012
45. 45
Phase 3 – Success Story….
First Win….and yes conversion had positive improvements…
Page Performance & Site Conversion – Feb 2012
46. 46
• Page speed matters for site conversion!
• Monitor real user performance in a “Big Data” way!!
• Every 1s improvement = Up to 2% increase in CVR
• 100ms improvement = Up to 1% incremental revenue
• SEO benefits for entry pages and reduce bounces
• Test & Learn - Target segments and run A/B Tests focused on
improving page performance
Key Highlights
Page Performance & Site Conversion – Feb 2012
47. We’re Hiring for Everything!!!!
47
Source: Flickr
http://www.walmartlabs.com/open-positions/
https://walmartstores.com/careers/apply/?ba=eCom
@cliffcrocker or @GoFastWeb