Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Similar a From good to great emetrics_dc_keynote
Similar a From good to great emetrics_dc_keynote (20)
Último
Último (20)
From good to great emetrics_dc_keynote
- 1. From Good to GreatHow the testing program is evolving at Dell Dell Online Analytics & Testing Team Oct. 6th, 2010
- 2. Agenda 2 What’s the vision? (Ed Wu – Senior Manager, Global Testing) Got process? (Carol Gohl – PMO) What’ve learned? (Dave Nelson – Global Testing Manager)
- 3. What’s the vision? 3
- 4. Online: Dell.com + Premier + Global Portal + B2B + Channel Portal + eSupport + Community 725K B2Btransactionsper year 34 supported languages Googled 1M timesper day 170M visitsto community properties 58Kchannelpartnersonline 1Bvisits per year to Dell.com 190KPremierPages 13Bpage viewsin FY’09 Dell Online Today 4 An online orderis placedevery2 seconds
- 5. Our journey started more than a decade ago… 5 Early Adoption Standardization Maturity Regional implementation Global implementation Standard KPI’s – global dashboard Regional KPI 1:1 BT experience Industry leading testing capability Internal testing capability Full funnel testing 2008-2009 2010- 1996-2007 In-house clickstream tool Advanced segmentation VOC solution Global team across BU Regional team Global team by BU Limited expertise and talent bench Work closely with vendors & consultants Grow in-house expertise Low
- 6. 6 Strong back-office support Good people Our Strengths Great technology Committed to testing
- 7. 7 “GOOD IS THE ENEMY OF GREAT” Jim Collins
- 8. 8 Dell.com Online Analytics, Testing & Targeting To be thetechnologye-commerce leaderby offering the mostrelevant experienceand solutions todrive high customerloyalty To become an innovation engine through insightful analytics &the best-in-class testingcapability to enable constant and rapid changes
- 9. People 9 Double the size of the team On-boarding process to provide support and create connections On-going training & development plan If I were running a company today, I would have one priority above all others: to acquire as many of the best people as I could. Jim Collins
- 10. Team Global Analytics Global Testing PMO eDell Programs Reporting 10
- 11. Accountability 11 2x Revenue upside Y/Y 400-600 tests globally Monetization model based on test results with statistical significance Complete transparency & alignment with finance & business Because they lack accountability, they fail to achieve credibility, and they have no authenticity. Jim Collins
- 12. Got Process? 12
- 13. Holy Grail of Web Analytics 13
- 14. Entering Carol Gohl… 10 years experience in PMO’s Information technology deployments Sales force transformation Construction and infrastructure Retail sales Product launches But, knew nothing about web analytics! 14
- 15. Key Findings in the First 30 Days Entrepreneurial vs organizational strategy Same people ….different roles! Multiple “unique” processes! Great testing ideas…capability challenges Investment in Analytics tools – but not process and workflow tools 15 Look… Listen… Learn…
- 16. 16 Planning cadence Pipeline management Standardized methodologies PRIORITIZE the elephants and the cows…. “We’ve never done that” “They” wont let you do that “That’s too risky” “We can’t do that”
- 17. Process Transformation Strategy 17 Architected Collaborative Faster deployment Near term benefit Not readily adopted Hard to sustain Slower deployment Lasting benefit More readily adopted Becomes organic
- 18. Big Picture First 18 pipeline roadmap Brainstorm Recipes Institutionalize learning implement Communicate Analytics Manager Lead Test Manager Lead Sign off required Documentation required
- 19. …and then get into Weeds 19
- 21. List of test ideas (NQ)
- 24. List of test ideas (NQ)
- 25. Annualized upside (NQ)Analytics Manager M3 testing priorities– wireframes / feasibility M2 testing priorities– wireframes / feasibility M1 testing priorities– wireframes / feasibility M1 testing priorities– wireframes / feasibility Idea Generation Lock M1 schedule; First test in M1 ready to launch Lock M2 schedule; First test in M2 ready to launch Lock M3 schedule; First test in M3 ready to launch Testing Manager Test Execution Weekly Optimization Meeting 20
- 26. Establish a Feedback Mechanism 21 Overall Satisfaction Improvement Areas
- 27. Process Change is Iterative…just like testing! 22 “Don't let the perfect be the enemy of the good” Voltaire
- 29. Focusing on Strategic Areas 24 Key Pages Cart and Checkout Targeting
- 30. Understanding Customer Intent 25 Specific product offers for customers desiring an efficient shopping experience Improved site navigation for customers want to take more time to find the right product
- 31. Targeting Relevant Content 26 Consumer Customers Small & Medium Business Customer
- 32. Checkout Testing - July Checkout Testing - August Checkout Testing - Sept CA Cons US Cons UK Cons US Cons DE Cons US SMB MX Cons CN Cons US Cons US Cons US SMB JP Cons AU Cons CN Cons CN SMB JP SMB AU SMB MX SMB BR Cons CA SMB US Cons US Cons AU Cons FR Cons US Cons MX SMB BR SMB US Cons AU SMB Testing Globally…an Iterative Process 27
- 33. Understand Why Something Works (or Doesn’t) 28 Simplified Cart Test – We changed too much vs.
- 34. Wrap it Up 29
- 35. Key Takeaways People first – invest more! Hold yourself accountable – be brave! Process is a necessity, not an option – start now! Scalability is key to success – leverage it! Failure is just as valuable as success – communicate it! 30
- 36. Embracing a New Culture:testing as a “religion” 31 Testing baked into dell.com projects from beginning Tests are run rigorously with clearly defined process Results captured and shared widely, good or bad
- 37. 32 “It took Einstein ten years of groping through the fog to get the theory of special relativity, and he was a bright guy.” Jim Collins
Notas del editor
- Customized Personal Experience: Not only personalization and behavioral targeting driven transactions, but… Customer choice focused ; Intent & context based Multi-dimensional ; Cross-platformCustomized social experience: Not only social media integration and ratings & reviews, but…Embedded in broader experienceContextual; Cross-platformCustomized Solution Experience: Not only rich product content and bundles, but… Configurable solutions; Global solutions development platform; Cloud
- Organizational design is about people firstGood people + robust process and the right tool sets – we had the people and the tools….
- No preconcieved notions – smart people - working hard – delivering results
- Running like small biz vs scaled org of a fortune 100 companyGeneralization of skills vs specialization of roles – lacks accountability/promotion of expertise and COCGeographic process disparity Dynamic funnel test ideas – but limited funnel accessRobust analytical tools ….and excel and powerpoint!Assemble the findings
- Identify the 5 big opportunities – the “elephants” in the room that if solved will bring the most immediate value to the orgTackle the obvious – its probably obvious to other too!Don’t pick and choose - No sacred cows - Listen for “we can’t change that” – “we’ve always done that” – “we can’t do that” – sometimes these are the hardest areas to affect and bring he most rewardChange the things that matter – go big or go home!Ask questions: Quantify and qualify the process opportunity – with the experts, the stakeholders AND the criticsCommunicatecommunicate - over communicate – change is scary Make process planning and change relevant – how will it benefit the team, or function - …it will make it more interesting to others!
- Organizationally decide how you want to change - Architected – 100 processes in 100 days – fast – quick resultsCollaborative – peel away the layers – slower, more painful, requires investment by allNo “not invented here”
- WARNING!Tests the sticktuitiveness of your team – process definition work can clear a room very quicklyFor key activities of each work area - answer the question: What, how, who and whenTrain on the key activities; Performance plans and quality plans should capture the required activities and deliverables
- Activities must align to a cadence or schedule (check durations or time to do the work)Define when the work starts and endsCheck process synergiesProcess is about who, what, how and WHEN
- Surveys, conversations and dataSet goals; measure to them – assess the effectiveness of the processesEstablish a feedback forum outside and inside your teamsExecute, checkand change
- a parting thought….In process design the 80/20 rule applies - Be secure in change control – it works for that 20%!Use the same discipline in rolling out a change in process as implementing a new process – changes are actually harder due to legacy knowledge
- When we formed our global team and got better at sharing information with each other, we started to learn things that have made us much better at getting the most out of our testing program. I’d like to take a few minutes to share with you some of the most important lessons we’ve learned.A while back, we weren’t very organized or focused, we just tested a lot of pages.We figured, the more pages we test, the more winners we’ll find, and we’ll improve our site faster. But we noticed that our most impactful tests throughout the world tended to be on the same pages – just a few for each country.We began to realize that we would be better off testing a few pages multiple times throughout the quarter than to test as many pages as possible.Another strategic focus area for us is basket and checkout pages.Even though these pages get a tenth of the volume of our most visited pages, they have the most revenue going through them. That shouldn’t be surprising because every purchase must go through these pages.The other reason these are valuable is that this is where we have are highest quality customers. This is especially true for Dell because we have a more detailed shopping experience than most stores. Many of our customers spend a lot of time choosing components and accessories to build the system that is exactly what they want. Customers who are willing to do this are seriously considering purchasing. Some of our greatest successes recently has been in cart and checkout tests.Our last focus area is behavioral targeting. The reason this is important to us is that we have a very high success rate with targeting.In fact, most of the targeting campaigns we try are successful.Whenever we focus on a customer’s needs and show them relevant content, they have a better experience and are more likely to buy.
- Its not enough just to know where on your site to test and which pages to focus on.You need to know what you should try – what you should change and how to change it.A common mistake by marketers is to think too much about what we want our customers to do. We’ll design a site around the way we want them to shop.It’s a good idea to turn that around and think about why your customer is on your site and exactly what they want to do. That’s an easy concept, but in reality it is very complicated because we all have so many different kinds of customers.A page that is designed perfectly for one customer might not meet the needs of another customer.Here’s one example of this from our site:Customers who are brand loyal and come to our site with the intention of purchasing expect a simple shopping experienceDon’t want a sales pitch, just want to see the product offersOther visitors are open to buying from Dell or one of our competitorsThey want to take their time making a decision and want reasons why Dell is better than our competitorsThey would prefer a site with clear navigation that makes it easy to find the information they are looking forIf we tested these two options against each other, one would likely come out on topSuppose the option on the left came out with a 3% advantage in revenue per visitorThat would be good, but doesn’t tell the whole storyIf we dig deeper we find that the option on the left works 10% better for Efficient Shoppers but 7% worse for our Cautious ShoppersThe best solution is to use a targeting campaign to provide the best experience to both group.Target Cautious Shoppers based on:Clicks on Help Me Choose and other learning contentView of customer reviewsVideosOther content that suggests a high level of customer engagement with the siteA great way to learn what works best for different types of customers is to do some segmentation analysis on your test results to see if the results vary for different kinds of customers.Sometimes a simple A/B test can result in a very successful targeting campaign. At the very least, you’ll learn more about your various customers, which will help you design better tests in the future.
- I wanted to show you a really simple of example of targeting we do to show our customers more relevant content based on what we know about them.This is our homepage in the United Kingdom. When a new customer comes to this page, they see a rotational banner with products across all of our segments, including consumer and business products.However, once a customer browses one of our customer segment sites, when they return to the country homepage, they will only see banners related to that segment.For example, if a visitor views pages in the Consumer site then returns, they will see banners featuring our consumer brands. Our Small and Medium Business customers will see products that would be relevant to them.We’ve done similar things with banners inside our customer segment sites. For example targeting specific product families based on prior browsing behavior.You should look to target based on any criteria you find that seems to differentiate the way customers shop. That might be clues from prior browsing history, traffic source, first time visitors vs repeat visitors, or based on prior purchases.The bottom line is, if we know what content would be relevant to a certain type of customer, we should use targeting campaigns to make them feel like the site is designed around their needs.
- One of the most significant changes to our test process is what we do with successful tests.Previously, if we had a successful test, we would implement the winner, have a short celebration, and then move on to some other idea.We now realize that we were missing out on a huge opportunity to drive more value out of every test we run.There are now two additional things we do every time we have a successful test.First, we take the things we learned from a test and attempt to design a new test that will perform even better.Even with a test that has positive lift, further analysis can reveal some things about the design that might not have worked like we intended. So, we’ll redesign the test and run it again to see if we can do even better.A good example of this was when we ran a series of tests related to adding a trustmark in our checkout pages.Our early tests revealed that we had an idea that worked, but we continued to test different trustmarks and different locations until we found a solution that worked best.The final solution had more than double the lift that we saw in our first successful trustmark test.The second thing we do with our winners is that we run similar tests in most or all of our key markets throughout the world. Once we found the best Trustmark solution in the US, we tested it in our other major markets.In essence, instead of running a bunch of stand-alone tests, we think of our testing program as an iterative process for providing a global solution.From a single test, we generate many new test ideas and designs, and we take our best ideas and test them throughout the world.
- Not only are we trying to get more out of our successful tests, but we now try to learn as much as possible from our mistakes as well.This happened in our checkout testing program when we tested the use of a simplified cart. We learned to not change too much at once.This slide illustrates the concept of the test.The hypothesis of this test was sound: We believed that customers would prefer a simplified cart with as little text as possible.So, we designed a test that removed or simplified any content we felt was unnecessary. Then we ran an A/B test.We were confident this test would work, so we were surprised when we found out the new recipe was performing worse than the original.It appeared that we removed some content that our customers thought was helpful. But we changed so much, how could we know which changes were positive and which were negative.Fortunately, in this case we got lucky. There was a change that caused significant negative performance that involved navigation, so we were able to detect it through some detailed pathing analysis of our test results. The change that hurt us was one of the smallest changes that we made, at least visually. We moved the Continue Shopping link from the top of the page next to the continue button to the very bottom of the page.This caused a frustrating shopping experience for customers who wanted to add other product to their cart, or go back to do more research before finalizing their order.We learned a couple of good lessons from this testFirst, if you are going to make a change that could significantly impact the way a customer browses your site – like changes to links to other pages, it is best to test those independently. Moving this link didn’t really fit with the original hypothesis of this test.Second, if you want to understand the impact of changing lots of different elements on the page, it is best to set up the test so that you can understand the impact of each change independently. Either multivariate test or an A, B, N test
- People firstTo have a successfultesting program, you’ll need good people, processes, and tools. Most important is people. Tools don’t give you anything on their own unless you have smart people to use them.Plus, smart people can help you define the best processes.Align people to tasks(Identify the tasks, then identify people to the tasks)(Specialists, accountability)Process is a necessity, not an optionFrom out experience we know that it is possible to run A/B tests without effective processes.However, you will need a streamlined process if you want to ensure you are consistently running the right test, executing them flawlessly, and maximizing the amount that you are learning from your testing program.Scalability is key to successEffective processes allow you to scale your testing program.Scale of people, tools, process, learningsFailure is as valuable as success
- In closing, I’d like to say that even if you have the right resources in place, the smartest people, and great processes, getting the most out of your testing program takes time. It will take a lot of failures and successes for you to learn the things you need to know to optimize your site. So be patient and stay committed to it, and it will definitely pay off.