Applying the science of measurement to the art of advertising - 20 january 2011 - slideshare
1. January 20, 2011 Ron Jacobs [email_address] Applying the Science of Measurement to the Art of Marketing Creating Meaning From Data, Analytics and Metrics A Presentation for
18. A/B Split Testing Headline Image Offer Body Copy Test A Test B Headline Image Offer Body Copy Call to Action Call to Action Test one variable at a time
19. Multivariate Testing Headline 1 Image A Offer 1 Body Copy a Test A Test B Headline 2 Image B Offer 2 Body Copy b Call to Action I Call to Action II Test C Head line 3 Image C Offer 2 Body Copy c Call to Action III Test multiple variables at a time 3 6 Variables (3x3x3x3x3x3) = 729 Tests
Users increasingly prefer different channels for different types of messages e.g. A phone app for checking airline schedules, web site for making reservations, SMS for flight delay notifications, email for upgrade and mileage status and Facebook feed for airline promotions. Each channels role in the marketing mix becomes clearer Channels have specific roles and message types Clarity allows marketers to better focus content creation on more meaningful messages; measure and evaluate success, and better communicate each channels benefits to management
33 one of the biggest problems most marketers have today is the short-termism that is killing businesses everywhere. When a customer has a good experience with your brand, say through an interesting visit to the store or a nice tour of a Web site, his likelihood of buying in the future increases, so his LTV increases. This increase in LTV represents the value created by that immediate experience. Therefore, LTV should be used not just to assess the attractiveness of prospects and prioritize your customer acquisition strategy, but it should be used by companies as a short-term metric of the long-term benefits of good (or bad) customer experiences!
1. Business contribution These are effectiveness measures comparing the performance of the online channel with other channels. Examples: • Online revenue contribution ($, %) – direct and indirect (i.e. transacted online and referred online from offline); • Online profit contribution ($, %) – Profit contribution to the company in the period; • Online sales transaction contribution (n, %) – direct and indirect (% sales online may differ considerably from % revenue or profit contribution if there is a different average order value or profitability online); • Online service transaction contribution (n, %, $) – what percentage of different types of customer service occur online. Cost savings can be calculated for these also; • Online reach % – Share of online users attracted to the site in an industry category in a week or month assessed by services such as Hitwise or Netratings. Strictly, reach should be assessed through reaching customers via third party sites; • Online market share – % of online market revenue captured in comparison with offline. This is difficult to establish in some markets, dependent on industry collaboration; • Online customer migration – % of existing customers using online services. 2. Marketing outcomes: • Sales (n, $) (If relevant); • Leads (n) (registrations of other opportunities to sell); • Cost per Acquisition (CPA) – Promotional cost of obtaining a first time sale; • Other costs – Cost of good sold and average margin. Cost of service; • Average order value (Basket size); • Lifetime value ($) for different customer groups; • Average touch frequency – for example, for e-mail marketing. 3. Customer satisfaction: • Customer satisfaction and loyalty indices; • Number of comments from site and e-mail (% favorable and unfavorable); • Brand metrics (brand favorability); • Site performance and availability; • E-mail enquiry response time and accuracy. 4. Customer behavior: • Site engagement rates (Bounce rates overall and for different pages); • Site conversion rates (Visit to Sale, Visit to opportunity and Opportunity to Sale); • E-mail conversion rates (Newsletter and campaign related); • Visits involving a page view in different categories (Product pages, Service pages, Where to Buy, Contact Us). Visits / customers can be scored according to this; • Visits to purchase/Time to purchase – indication of number of visits involved with purchase; • Number of products purchased per customer; • Transaction behavior (Recency, Frequency, Monetary value analysis for different categories and customer types). RF analysis also relevant for site visits, e-mail response and different service types; • Activity or participation levels (Percentage of customer base / registrations who are actively using online service(s)). nActivated, nActive, nDormant, nLapsed, etc.; • Loyalty or churn metrics (% of customers repeat purchasing in given time, e.g. 1 year). 5. Channel promotion: • Referrer mix from different sources (direct, search, affiliates, etc); • Share of search (main terms within market); • Cost Per Click/Cost Per Contact (Visitors) average and CPM average for online/offline ads. 6. Social Media A variety of measures that quantify some very soft measurements. This is state of the art just now.
Like other forms of direct marketing testing, merchants can use either A/B or Multivariate testing to test headlines, offers body copy, offers, calls to action, images, product pages, order pages, guarantees, prices, bonus offers, and many other variables Merchants often report dramatic increases when testing different copy text, form layouts, landing pages, images and background colors. However, not all elements produce the same increase in conversions, and by looking at the results from different tests, it is possible to identify those elements that consistently produce the greatest increase in conversions for a marketer’s web site. Different customer groups react differently, which is why testing is so important.
There are two kinds of web testing for E-Commerce sites. A/B testing, also called split testing, allows the testing of two different versions of a design, copy or offer, to see which performs the best. For decades, this has been a classic method in direct mail, where companies often split their mailing lists and send out different versions of a mailing to different recipients. A/B testing is also popular on the Web, where it's easy to make your site show different page versions to different visitors. Use A/B testing when a web site gets fewer than 1,000 page views per week. It is useful when testing big things. For example, if moving complete sections provides an advantage or if changing the overall copy and design works better than the established copy and design. Multivariate testing is the most robust way to test a lot of variables at one time. This advanced statistical methodology can test the effectiveness of limitless combinations. The only limits on the number of combinations and the number of variables in a multivariate test are the amount of time it will take to get a statistically valid sample of visitors and a marketer’s computational power. This form of testing can only be done when a web site receives more than 1,000 page views per week. It can help an E-Commerce merchant optimize multiple content changes in different parts of a multiple web pages simultaneously.
A/B testing is the simplest and easiest form of web testing. Based on “the Scientific Method” of statistical analysis, it’s main limitation is that only way variable can be tested at one time. In the example, all variables remain the same with the exception of the design. So, design is the variable that was tested. Headlines, offers, images or products could be tested. Often merchants choose A/B testing when they want to test completely different pages. That’s okay, so long as what is being analyzed as the variable is the complete difference. Like other forms of direct marketing, it’s best to test the big things. If you can’t see the difference in two different combinations being tested, it’s unlikely that visitors will.
Multivariate testing is an area of high growth, as it helps websites ensure that they are getting the most from the visitors arriving at their site. Search engine optimization and pay per click advertising bring visitors to a site and have been extensively used by many marketers. Multivariate testing allows marketers to ensure that visitors arriving at their website are being shown the right offers, content and layout to convert them to sale, registration or the desired action. In the example, headlines, offers, copy, images, background colors are tested. It’s easy to get carried away with multivariate analysis. So, it’s better to test a small number of variations to insure that there are at least 100 conversions per combination analyzed. Website visitors will vote with their clicks for which content they prefer. Multivariate testing is transparent to the visitor, and technology is capable of ensuring that each visitor is shown the same content on every visit. Both A/B and Multivariate Testing allow the market to decide a marketer’s best web page options. There is no room for guessing.
Most current attribution models are flawed The last touch standard or current session models Causes over-invest in near-term conversion drivers The consumption and impact of media is interrelated with other media Traditional media is interrelated with Digital media The relationship between display and search changes depending on products, brands, time of day, season, company, geography, etc. The multichannel effect is important in high-consideration situations e.g. Expensive, complex or involved offerings Financial services offerings, family vacation or a choice of college
Exploring The Social Compass A compass is a device for discovering orientation and serves as a true indicator of physical direction. Inspired by a moral compass, The Social Compass [10] serves as our value system when defining our program activities. It points a brand in a physical and experiential direction to genuinely and effectively connect with customers, peers, and influencers, where they interact and seek guidance online. It was designed to guide us from the center outward. However, it can also impact how a business learns and adapts by reversing the process and listening to customers and influencers through each channel from the outside in.
Overall, every area of social media strategy will see more budget increases than decreases in 2011. As consumers and influencers continue to flock to social media—and social media programs and marketing demand more resources and budgets—demonstrating ROI for those who determine said budgets is key. Additionally, getting buy-in from stakeholders and increasing budget and headcount are also popular objectives, with 32.2% and 24.6% of respondents reporting such goals for 2011, respectively. Determining ways to measure ROI and demonstrating how social media can provide direct value and results will help reach several of these additional objectives.
Social Engagement Index (SEI) - The SEI is a proxy for a brand social reach and is calculated by weighting the raw number of conversations by the reach of its participants. The raw score is then calibrated into an index. A score of 100 is the base brand score. Anything above this indicates a greater net reach of social conversations compared to the average brand. Social Sentiment Engagement Index (SSEI) - The SSEI is a composite that combines measures of both engagement and sentiment. We calculate engagement by measuring the raw number of social conversations factored upon the reach per conversation participant. We then apply a function that accounts for the sentiment of positive and negative comments. Finally we calibrate this into an index based upon 100 point brand score. Anything above this indicates a greater net amount of positive engagements, while a score less than indicates more negative. The further away from 100 a score falls the more intense the sentiment.
Social Sentiment Engagement Index (SSEI) - The SSEI is a composite that combines measures of both engagement and sentiment. We calculate engagement by measuring the raw number of social conversations factored upon the reach per conversation participant. We then apply a function that accounts for the sentiment of positive and negative comments. Finally we calibrate this into an index based upon 100 point brand score. Anything above this indicates a greater net amount of positive engagements, while a score less than indicates more negative. The further away from 100 a score falls the more intense the sentiment.
Cost Per Social Impression (CPSM) - How much would you be willing to pay for a Tweet? or a new fan or follower? Clearly social media is in its infancy as a cross-channel media measurement tool, but already it's clear the social space is an excellent medium for measurement as it reflects and resonates brands spend in other channels. In an effort to gauge how successful the brands were at converting their Super Bowl media spend to social engagement we've taken the potential reach of the conversation, using a popularity score as a multiplier, and divided it by the media spend. In looking at a brands CPSM, the closer to $0.00 the better.