1. eCommerce Performance
What is it costing you, and what
can you do about it?
Peter Holditch
Technologist
pholditch@appdynamics.com
2. The Business Impact of One Second
“One second increase in
Amazon‟s page load
would annually cost $1.6
billion in sales”
Borland Research - March 2013
3. Because a 1 second delay equates to…
A 16% decrease in customer satisfaction
11% fewer page views
A 7% loss in conversions
3
4. Google and Microsoft research
• Experiments to introduce delay into web
searches to measure the impact
http://velocityconf.com/velocity2009/public/schedule/detail/8523
http://vimeo.com/5310021
4
5. 50ms
200ms
-0.3% -0.4%
500ms
-0.6% -1.2% -1.0% -0.9%
1000ms -0.7% -0.9% -2.8% -1.9% -1.6%
2000ms -1.8% -2.1% -4.3% -4.4% -3.8%
- Means no statistically significant change
li
cre
ase ck
in m
s)
oC
(in
et
Tim
on
isfa
c ti
Sat
Dis
t in
Qu c t
e ri
es/
Us
Qu
er
e ry
Re
fin
em
en
Re
t
ven
ue
/U
ser
An
yC
lick
s
Server Delays Experiment: Results
500
1200
1900
3100
• Strong negative impacts
• Roughly linear changes with increasing delay
• Time to Click changed by roughly double the delay
6. Impact measured by
• Slower performance abandoned searches
• More active users more sensitive to this
• Effect got worse over time, and persisted once
performance was restored
wk1
wk2
delay
removed
0.2%
0%
200 ms delay
400 ms delay
-1%
-0.8% -0.6% -0.4% -0.2%
0%
-0.8% -0.6% -0.4% -0.2%
actual
trend
daily searches per user relative to control
0.2%
Persistent Impact of Post-header Delay
200 ms delay
400 ms delay
-1%
daily searches per user relative to control
Impact of Post-header Delays Over Time
wk3
wk4
wk5
wk6
actual
trend
wk3
wk4
wk5
wk6
wk7
wk8
wk9
wk10
wk11
6
7. Conclusion
• Revenue is a function of user behaviour
• User behaviour is quite sensitive to
performance
• Effects of poor performance outlast the
problems
• It is necessary to have a constant watch on
performance of critical transactions, fix
problems quickly and continuously improve
over time
7
8. This is made very hard by the modern technology landscape
Distributed
Monolithic
Release 1.1
Release 1.2
Release 1.23
Tomcat Release 1.5
.NET
Amazon EC2
Windows Azure
CLOUD
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Login
Search Flight
View Flight Status
Make Reservation
Tomcat
Mule, Tibco, AG
Tomcat
ESB
VMWare
WEB 2.0
Memcached
Oracle
Weblogic
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Browser Logic
AJAX
Web Frameworks
Coherence
Hadoop
Cassandra
MongoDB
SOA
.NET
MQ
AGILE
Release 3.4
Release 3.5
Release 3.6
Release 4.0
SQL
Server
Release 4.4
Release 4.5
Release 4.6
Release 5.0
JBoss
Release 1.4
Release 1.5
Release 1.6
Release 2.0
BIG DATA
ATG, Vignette,
Sharepoint
8
9. Where and what is the problem?
Release 1.1
Release 1.2
Release 1.23
Tomcat Release 1.5
.NET
Amazon EC2
Windows Azure
CLOUD
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Login
Search Flight
View Flight Status
Make Reservation
Tomcat
Mule, Tibco, AG
Tomcat
ESB
VMWare
WEB 2.0
Memcached
Oracle
Weblogic
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Browser Logic
AJAX
Web Frameworks
Coherence
Hadoop
Cassandra
MongoDB
SOA
.NET
MQ
AGILE
Release 3.4
Release 3.5
Release 3.6
Release 4.0
SQL
Server
Release 4.4
Release 4.5
Release 4.6
Release 5.0
JBoss
Release 1.4
Release 1.5
Release 1.6
Release 2.0
BIG DATA
ATG, Vignette,
Sharepoint
9
10. Where and what is the problem?
Release 1.1
Release 1.2
Release 1.23
Tomcat Release 1.5
.NET
Amazon EC2
Windows Azure
CLOUD
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Login
Search Flight
View Flight Status
Make Reservation
Tomcat
Mule, Tibco, AG
Tomcat
ESB
VMWare
WEB 2.0
Memcached
Oracle
Weblogic
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Browser Logic
AJAX
Web Frameworks
Coherence
Hadoop
Cassandra
MongoDB
SOA
.NET
MQ
AGILE
Release 3.4
Release 3.5
Release 3.6
Release 4.0
SQL
Server
Release 4.4
Release 4.5
Release 4.6
Release 5.0
JBoss
Release 1.4
Release 1.5
Release 1.6
Release 2.0
BIG DATA
ATG, Vignette,
Sharepoint
10
11. What if the problem is outside the application?
Release 1.1
Release 1.2
Release 1.23
Tomcat Release 1.5
.NET
Amazon EC2
Windows Azure
CLOUD
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Login
Search Flight
View Flight Status
Make Reservation
Tomcat
Mule, Tibco, AG
Tomcat
ESB
VMWare
WEB 2.0
Memcached
Oracle
Weblogic
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Browser Logic
AJAX
Web Frameworks
Coherence
Hadoop
Cassandra
MongoDB
SOA
.NET
MQ
AGILE
Release 3.4
Release 3.5
Release 3.6
Release 4.0
SQL
Server
Release 4.4
Release 4.5
Release 4.6
Release 5.0
JBoss
Release 1.4
Release 1.5
Release 1.6
Release 2.0
BIG DATA
ATG, Vignette,
Sharepoint
11
12. Real-User Monitoring gets Real Results*
>91% transaction
completion
End-users
„completely satisfied‟
12
*Source: Aberdeen Group, July 2012
>30% increase in
App Availability
Businesses NOT doing Real User
Monitoring
Businesses doing Real User
Monitoring
>10% decrease in
end-user complaints
13. And beyond performance monitoring…
Release 1.1
Release 1.2
Release 1.23
Tomcat Release 1.5
.NET
Amazon EC2
Windows Azure
CLOUD
Release 2.4
Release 2.5
Release 2.6
Release 3.0
Login
Search Flight
View Flight Status
Make Reservation
Tomcat
Mule, Tibco, AG
Tomcat
ESB
VMWare
WEB 2.0
Memcached
Oracle
Weblogic
Release 1.4
Release 1.5
Release 1.6
Release 2.0
Browser Logic
AJAX
Web Frameworks
Coherence
Hadoop
Cassandra
MongoDB
SOA
.NET
MQ
AGILE
Release 3.4
Release 3.5
Release 3.6
Release 4.0
SQL
Server
Release 4.4
Release 4.5
Release 4.6
Release 5.0
JBoss
Release 1.4
Release 1.5
Release 1.6
Release 2.0
BIG DATA
ATG, Vignette,
Sharepoint
13
14. Case Study – One Year
Pre-Production
Dev
Production
QA
• Agile Releases 12 > 18
• Spent 3,060 hours less firefighting
• Delivered More Innovation
Ops
Business
• Availability 99.91% > 99.95%
• MTTR 40 hours > 22 hours
• 1,528 hours less troubleshooting
• Identify & Fix Defect 20 hours > 13 hours
• Spent 4,024 hours less testing
• Faster Time to Market
•
•
•
•
End User Experience 500ms > 150ms
$167,475 lost revenue savings
$627,691 productivity savings
$795,166 Total savings
14
15. Want to learn more?
Visit the AppDynamics Booth
Near the eCommerce Theatre and Mobile &
Social Media World Theatre
Download free monitoring solution:
Bit.ly/IWLite
Follow @AppDynamics
15
A study by Borland identified an overwhelming correlation between sales-generated traffic rises and increases in website response times – a nightmare situation for any retailer hoping to capitalize on the seasonal online rush of bargain-hunting consumers. Research has shown that even minor delays to website response times can have a sizable impact on customer satisfaction, page views, conversion rates and site abandonment. A one second delay in website response time equals11% fewer page views,16% decrease in customer satisfaction and a 7% loss in conversions.The study thus concludes that a one second increase in Amazon’s page load would annually cost $1.6 billion in sales, and 38% of UK online shoppers abandon websites or apps that take more than 10 seconds to load.The average online shopper expects web pages to load in 2 seconds or less, after 3 seconds, up to 40% will abandon the site. Seventy four per cent of users will abandon a mobile site after waiting only five seconds for it to load.Once visitors leave, it’s very difficult to get them back. 88% of online consumers are less likely to return to a site after a bad experience.Play.com, the UK arm of the Rakuten Group, saw performance drop by 500% as its site slowed from a load time of 2 seconds to 12 when site traffic peaked on the 4th January. Other online retailers that also suffered significant increases in load times during the first few days of the January sales included John Lewis, Amazon.co.uk, Asos.com and Tesco.com. Increases ranged between 3 and 4.5 seconds for their landing page to load.“There is lots of data available showing that users are losing patience with poor performing websites,” said Archie Roboostoff, product director at Borland. “It looks like a number of the sites monitored over the seasonal period will have missed out on potential revenue as a result of their website’s inability to process high levels of traffic. The sites we monitored in the UK had normal load times averaging 2.9 seconds, but saw load times increase by an average of 4.5 seconds during peak traffic periods – a 55% deterioration.Developing a robust performance strategy takes time, and peak period preparation should begin early with testing starting about six months beforehand. Putting in this groundwork is crucial if retailers are to take full advantage of peak shopping times throughout the year.”http://www.retail-digital.com/retail_technology/one-second-delay-on-amazon-16-billion-loss-a-year[source data: http://www.aberdeen.com/aberdeen-library/5136/RA-performance-web-application.aspx]
The application landscape is complex, and so is the transaction landscapeSome transactions will be more important to track than others – with conventional monitoring it’s impossible to focus on the important things, and impossible to understand if monitoring anomalies have any business impactMoreover, it’s impossible to troubleshoot the important things – just
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Find the point of a problem quicklyGather enough detail to troubleshoot it in situDo the same during development, to avoid issues getting to production
Objective of SlideHighlight our value proposition across Development, QA, Operations and the business.ScriptFor example, here’s a customer case study from Edmunds.com which highlights the annual benefits of AppDynamics across their organization and lifecycle.Development was able to double their innovation as a result of spending less time firefighting, and implementing more business requirements.QA were able to detect performance defects twice as fast, therefore increasing testing productivity and accelerating time to market.Operations increased application availability by .04%, and cut MTTR in half which had a significant impact on the business.All these benefits translated an enhanced end user experience combined with significant lost revenue and productivity annual savings totaling almost $800,000.Bank of New Zealand, Expedia and Fox News also had similar savings to Edmunds.com.
Objective of SlideHighlight our value proposition across Development, QA, Operations and the business.ScriptFor example, here’s a customer case study from Edmunds.com which highlights the annual benefits of AppDynamics across their organization and lifecycle.Development was able to double their innovation as a result of spending less time firefighting, and implementing more business requirements.QA were able to detect performance defects twice as fast, therefore increasing testing productivity and accelerating time to market.Operations increased application availability by .04%, and cut MTTR in half which had a significant impact on the business.All these benefits translated an enhanced end user experience combined with significant lost revenue and productivity annual savings totaling almost $800,000.Bank of New Zealand, Expedia and Fox News also had similar savings to Edmunds.com.