Analytics, if done well, produces insights, but insights on their own produce nothing. USAA has integrated their analytics streams with their marketing strategy and execution streams. In this session Robert Wellborn presents insights and finding they learned on the way to integration. These include: aligning marketing with the bill payers; gaining the right sponsors; and structuring your organization to generate and consume insight.
Generative AI for Technical Writer or Information Developers
Optimize Your Marketing Mix
1. Optimizing Your Marketing Mix
A case study
May 1, 2014
Robert Welborn
Associate Vice President, Decision Scientist
USAA
@robert_welborn
2. Cautionary Tale
• In 1949 Alexander Kotov was challenge by Tigran
Petrosian (The Iron Tigran)
•Petrosian is the acknowledged Defensive Master,
able to see 1,000’s of variations in play. He is the
“hardest to defeat player in the history of chess”
• Kotov on the other hand was the chess rockstar.
He hobnobbed with supermodels and nuclear
physicists.
•This was Superman fighting The Incredible Hulk in
the Soviet Chess world
•Petrosian surrendered in 13 moves
Source: Wikimedia Commons
3. Kotov Syndrome
• A Player, confronted by 1,000’s of possible scenarios
• The obvious is missed
• The player makes decisions based on emotion and
expediency, instead of facts and discipline
4. Industrial Analysis requires Decision Frameworks
not just Decision Making Tools
Analytics must produce Insights, not reports, and
the Insights must be Useful (Results are better than
elegance)
The Usefulness of the insights is a function of the
integration of your analysts (not just your analysis)
Three Keys
5. Building Decision Frameworks
(not just buy more tools)
Organize To Use Specialists
Start with the CFO
Publish for the C-suite
Source: Beleant, 2007 “Paleolithic Flints” used under Michigan
Publishing Creative Commons http://hdl.handle.net/2027/spo.7523862.0005.006
6. When to Pay Attention to Analytics
The General Practitioner versus the Specialist
• When the GP says you are sick, you trust the GP to run some
tests
• When the specialist says that you have Achilles tendonopathy,
you go have surgery
•You want your analysts to be specialists
Source: Wikimedia Commons
7. So Who are The Specialists
The Threefold Path of Analysis
Ask someone
Experiment
Build a model
8. The CFO wants marketing to work
She may not know why
Or how it works
•
Transform the CMO into a portfolio
manager
•
Analytics becomes the risk manager
9. Publish for the C-suite
Connect competitor action and macroeconomic analysis to your marketing
Allow real scenario analysis, "what if"
Get out of the PowerPoint trap and into white papers and long reads
Become the CEO's weekend reading
Source: Longreads.com
10. Analytics Must Produce Insights
(Focus on Results)
• The Results should drive where
you start, remain, and stop looking
• Converting Invested Money into
Products is the kernel of every
useful insight in Marketing
11. Insights must be useful
• The more useful the insights, the
more credible your analysts are
• The more
http://blog.uber.com/wp-content/uploads/2011/12/New-Logo-Vertical-Dark.jpghttp://blog.uber.com/wp-content/uploads/2011/12/New-Logo-Vertical-Dark.jpg
12. Conclusion
• Align your organization for using specialists, and set them up so
that there is no punishment for telling the truth
• You need to hire the following specialists: Survey-ers,
Experimenters, and Modelers
• Connect the stuff the CFO wants to what the CMO delivers and
manage the risk
• Publish for the C-Suite and use your credibility
• When you are certain of something, do something
18. Separate the method of marketing from
the method of measurement
Certainty in a measurement is not the same as certainty in results
Paid Search vs. Lift over control
I'm here today to talk to you about how to do something with the stuff that you know. This is a case study involving USAA and other companies, and I want you to understand what works and what does not.
Kotov was the Russian Austin Powers. He wrote books, hung out with spies and bragged about the superiority of Soviet chess players
You can have all the metrics in the world at your finger tips but you have to do something or stop something with what you know from your metrics
The more data points, analysis, and insights. Analytics produces insights, some are useful, some are not. The more general the application of the insight, like knowing every opening and defense for chess. One large exception, though, You can string together detailed analysis from the bottom up to get very useful broad interpretation, Consider Nate Silver's approach versus the Washington Redskins approach.
Imagine what happens to you when you are confronted with Market Scenarios, Competitor Scenarios, Macroeconomic scenarios, Media Buying deadlines, Emerging Media Options, Agency Meetings, 30 second spots versus two split 15
Oh, but what are the right topics?
That brings me to the point of my presentation. Seriously, I'm 6 slides in and just now telling you what I'm going to talk about. The thing that USAA did that worked so well was that our Marketing, Analytics, Research, CFO and IT organizations all organized themselves around the idea of using Specialists.
Really, that's it. If you get nothing else from our discussion it is this, you can hire specialists, you can let them do research, you can look at their results, but if you are not organized that when a specialist produces a result, you use it or start to use it immediately, you will waste your time.
So 6 slides later, now let's talk about how to organize your business to use specialists
If your GP says that you stage 4 ulnar encepholopathy, you would rightly ask if he would like to run some tests or send you to a specialist. This is always true if you can't pronouce your disease or point to it on one of the creepy charts on the wall. You trust the GP to run tests and to recommend. You seek out a specialist for everything else.
If you hire generalists, what you will get is the opportunity to do more research, and research begets research, in a never ending spiral, called analysis paralysis
You want to hire, train, groom, promote, cajole, and whatnot your way to analysts who are specialists in the right topics
So where do you need specialists?
There are really only three ways to analyze anything
In many organizations the CFO and CMO are at odds with each other.
The CFO's goals showing tangible returns quickly.
The CMO has to look at customers like a farmer, there is a time to plant and a time to sow. Sometimes the objective of a campaign is to increase awarness, sometimes the objective is increase sales. That may be interesting, but what the CFO really wants to know is where she can measure money going out compared to money coming in. So often the CMO and the CFO struggle to find a common measurement framework
Enter Analytics, with structural equation models, multivariate experiments and other methods, now we can connect the dots between awareness and products. The first place that analytics has to produce results is in translating the work of the CMO into a common vernacular for the CFO. It allows for a new role for the CMO as the CFO's portfolio manager with Marketing Managed budgets.
Analytics can tell the CMO and CFO that if you raise awareness 5% you should see a 3% increase in sales.
More importantly, for me, it allows Analytics to be the "there are too many eggs in this basket role". Which sounds more sophisticated when you call it, Risk Manager.
The real secret to getting your analytics used, is to make them useful.
Building a good reputation in the C-Suite Inspires action
Unsolicited, we started publishing a Weekly Marketing Estimate Update. It bascially takes all of our analysis, roles it together, and says this is how many product you are going to get next week, next month, until the end of the year, and what marketing efforts are driving each product.
Second, we started publishing scenarioario analysis, working with our risk and economics folks and saying what would happen if we had a second government shutdown, and how long is "too long". This takes us from looking in the rearview mirror to looking through the windshield. Everybody knows Marketers steer better when looking through the windshield.
Third, the Powerpoint trap is turning senior management into third graders. These are smart people. You didn't need to explain everyting in 3 bullet points when they were VP's and asked informed questions. Yes, they are busy, This is why I really like Longreads. It's a website that is focused on long form journalism with interactive visuals. You actually know something new after reading one of these stories. Think "The Atlantic" meets crowd sourceing. It's the anti-Twitter, and thats from someone who loves Twitter.
You need to give the CEO something worth reading and learning, Something that they can skim and understand or take half hour to an hour and truly be up to speed.
I was once on a team with two paid search analysts. During the course of their first meeting with me, I was their peer, they explained that they were terrified because they had no evidence their paid search campaigns were working. I looked at their method, we restructured their tests and determined, their paid search campaigns really did not work. The first executive that I showed this too took me aside and said, "Are you crazy, I'm going to have to fire these two analysts, and stop doing paid search. Come back with a more optimistic analysis of paid search."
Optimistic Analytics do not exist. Optimistic results occur occasionally. There is a reason economics is called the dismal science.
USAA created an independent and impartial Chief Data and Analytics office, where we keep Measure seperate from Recommend. Measure as in campaign results and recommend like proposed marketing mix, and tactics. If you recommended something that didn't work, someone else, holds you accountable. If you measure the wrong thing or the wrong way, Recommend holds you accountable
The last point is a bit finer. Research, the traditional research with marketing folks in lab coats questioning customers on how they feel when they see 237 shades of orange or thematic implications of infintessimal pertubations of sparse matrices of on driver habits, really needs to be seperate from analytics. For the most part we are working with data generated around actual implemenation of plans, while traditional research is working with theoretical data.
Specialists in survey need to understand
More importantly they need to be able to produce testable hypotheses that the experimenters and models can prove out
In return, the Survey teams need to be able to validate models and experiments
Experiments were once the arcane practices of direct mail archmages coming out of their cubes between sessions of Dungeons & Dragons to hand the marketing team an audience list,
A guy at MIT once told me Experiments take you from correlation to causation.
The skill has advanced and now it's heralded by terms like interactive and geospatial. The focus is still on addressable markets, places where you know the name or address of the individual involved in the experiements, but tools like Google paid search with geotargeting allow you to hold out samples that are seperated by a few blocks, instead of entire markets
It's not just testing one thing anymore, the multivariate approach allows for partial control groups, multiple use cases
However, the experiments can be confirmed with surveys and models, so you can grade experiment effectiveness
The point is that you can at any time, test your method of analysis, and you should. To the point that you trust your analytics to execute on them immediately. That's certainty in measurement. You need to be certain that your method of measurement is right, not that the results will be right. Sometimes the Government shuts down during your new product launch, sometimes, your ad goes viral and way more people than you every planed see it.
Second, you do not need a paid search measurement analyst or an email analyst or a direct mail analyst. You need analysts who understand paid search and who understands design of experiments and web analytics. Their toolset has to reach beyond the marketing method. If your email analyst figures out email doesn't work, is it his fault that its not working? There is no reason to put the analysts in the position that they are rewarded for concealling or embellishing the truth.
When your are certain in your method of measurement and aligned toward your method, Then you are ready to go to the CFO.
This is the one that I do for a living, so of course this is the one that I include a quote on.
Pablo Picasso was of course not speaking about living breathing people employed as models.
In marketing you will build abstractions of reality to determine propensity to consume, the yield of a campaign, the rate at which people convert from prospects to consumer to lost customers
The important part here is that if your model says Christina over here will buy the product you can a) call Christina and ask her if she is interested in the product or b) send Christina a message, through the mail, her Twitter feed, on a banner ad, and see if she or people like her buy at the rate you expect